Jump to content
View in the app

A better way to browse. Learn more.

Surly Horns

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.
Football ... Basketball ... Baseball ... Other Sports ... Futbol ... 🤫995🤫 ... Gambling ... Movies & TV ... Music ... Hobbies ... Lulz ... Food & Travel ... Daily Texan ... Business & Markets ... Cloak Room ... Help ... For Sale ... Board Discussion ... Advertise... Tailgate Donations

Featured Replies

#5701
42 minutes ago, immamac said:

Chicago latency from Texas is lowest and geographically far enough to where the same event is unlikely to have effected both. 

Florida has too many hurricanes. North Carolina didn't have any capacity, didn't wanna do Arizona, but could look into it. California or any west region is too high latency for me same with the full on northeast regions. 

Surly is on the backbone, which is one of the reasons we don't have to have a cdn to serve content surly gets to the open web in 1 hop, through the redundant interconnect switch. Surly for users in Texas is actually faster than the majority of CDNs. For users outside of Texas it's likely just as fast. When I travel I test it pretty thoroughly and the latency and bandwidth is totally fine. 

I was only considering locations on backbone and not buried behind 2-3 interconnects to get to there. 

I may set up a data replication service off backbone to make DR trivial, the hard part right now is dealing with data syncing between sites as replication is one way by design right now. 

This was a catastrophic event and all system durability was proven pretty well which i am happy about. There has never been a time in this current Datacenter where we have had complete power loss. We had a network hiccup a few years ago due to a misconfigured switch but that was a blip in comparison to today's real deal full dark outage. 

You're good, dude. My post wasn't intended as criticism of you or this site. I just wanted to diss Chicago.

Edited by Braff Zacklin
wasn't

  • Replies 7.1k
  • Views 469.2k
  • Created
  • Last Reply

Top Posters In This Topic

Most Popular Posts

  • Tornado hit data center. Sorry for downtime. Going to have more downtime to fix later Please report weird shit here but I'm with my family and mobile only. I just spent about an hour on a bench o

  • hookemATL
    hookemATL

    I had to draw porno. Wasn’t great but I finished

  • https://www.surlyhorns.com/board/settings/account-security/#piiDataRequest if you want to know what data could have possibly been leaked request here.  The passwords are encrypted at rest in

Posted Images

#5702
For non nerd version. I only put equipment I own on the red dots which are very specific facilities. Choosing to do it this way makes it cost effective and the most robust and reliable deployment possible. 
image.png.c925bf747f81f238e2cafdb1a0a3ca0f.png

@terriblemaps
#5703

One of the sites I like (killsixbilliondemons.com) is hosted by NixiHost, whose servers are in the DFW area. They were also out this morning, but are back up now.

#5704
57 minutes ago, immamac said:

For non nerd version. I only put equipment I own on the red dots which are very specific facilities. Choosing to do it this way makes it cost effective and the most robust and reliable deployment possible. 

image.png

Why do you hate South Dakota and New Mexico. The Surly population in those meccas of assholes is massive. 

#5705
1 hour ago, immamac said:

I was only considering locations on backbone and not buried behind 2-3 interconnects to get to there. 

I may set up a data replication service off backbone to make DR trivial, the hard part right now is dealing with data syncing between sites as replication is one way by design right now. 

IMG_2754.jpeg.bcb487906893d323118c0dd54994ba6a.jpeg

#5710
5 hours ago, Hornius Emeritus said:


Don't worry about it. Shit happens. You don't owe us anything.

However, I'm about to get evil on those assholes in the 2025 recruiting thread who insist on discussing whether Katy is in Houston. Jeeze.

Of course Katy is Houston. 

#5711
5 hours ago, immamac said:

For non nerd version. I only put equipment I own on the red dots which are very specific facilities. Choosing to do it this way makes it cost effective and the most robust and reliable deployment possible. 

image.png

rh1dnt91gtaz.gif

#5712

screw fixing search

it's so bad, just turn it off

kill it with fire

is there a way you can macgyver the search box to register as an ad impression and redirect to google?

#5713
10 hours ago, immamac said:

I was trying to get shit back up with screaming children doing the dino exhibit at Lady bird wildflower center. 

Fucking legend. 

#5716

Was Texassports.com being all borked yesterday caused by the same issue? It was a really weird feeling when two of my most accessed sites were down. And I’m currently in the middle of The Last Stand book series which is about an EMP blast, so my mind went to some weird places for a bit.

#5718
44 minutes ago, SquishMitten said:

Was Texassports.com being all borked yesterday caused by the same issue? It was a really weird feeling when two of my most accessed sites were down. And I’m currently in the middle of The Last Stand book series which is about an EMP blast, so my mind went to some weird places for a bit.

no idea, but yes this took a lot of sites down it's not some shitty little third rate colo.

#5719
On 5/26/2024 at 11:22 AM, Buzzrock said:

You should try the cloud. Clouds don’t get hit by tornadoes. 

 

On 5/26/2024 at 11:29 AM, Gatorubet said:

IMG_2749.jpeg.40a56145c6b7048fabb1daf33d8492c0.jpeg

The cloud is just someone elses' computers.

FIFY.

But, @immamac, have you considered going active / active, and doubling up the A records in DNS so you can distribute traffic accordingly?  Or would the sync times between the two DCs just kill that idea?

#5720
28 minutes ago, PvilleStang said:

 

FIFY.

But, @immamac, have you considered going active / active, and doubling up the A records in DNS so you can distribute traffic accordingly?  Or would the sync times between the two DCs just kill that idea?

Why? It introduces so much complexity for basically no gain. 

 

If surly produced millions a year in revenue and did hundreds of financial transactions a second I'd do some of this stuff, it's just not really worth it or feasible for the level of revenue. 

#5721

I didn't realize surly was rawdogging it to the internet with no CDN, that definitely makes the fail over and fail back much more complex. But on the other hand, if you rarely ever use the DR resources because of the hassle to change over, what's the point in maintaining it? Why not just do change data capture for the posts to the off-site and rebuild when needed?

 

#5722
34 minutes ago, Captainant said:

I didn't realize surly was rawdogging it to the internet with no CDN, that definitely makes the fail over and fail back much more complex. But on the other hand, if you rarely ever use the DR resources because of the hassle to change over, what's the point in maintaining it? Why not just do change data capture for the posts to the off-site and rebuild when needed?

 

Why do that when you can just replicate off site? DR site is exactly that. A site that can scale up in the event of DR to serve traffic, it still has to have a complete copy of the data there. The data hotel I was talking about setting up as a tertiary never to serve up anything place is what you are suggesting, which isn't a bad idea I just don't know if it fucking matters. 

The reality is it wouldn't have been worth flipping to DR then having to wait till night to take the downtime and switch back to main and have circular replication or an extra redirect for another day because DNS propagation is potato even with 60s TTL. I'd rather just take the downtime up front if it's only an hour or 2 than to spend 12-16 hours of my own time making sure shit goes well and is fine.

Posts are only one part of the whole thing that makes the site go. 

Check this out. I had to use a Dallas DB server that I replicate to just in case some shit happens in the rack to serve traffic for the day. This is what that looks like. Sustained 300+Mbps and averaging well into the 400+Mbps range. Just for the database. I think people fail to recognize how big the site is constantly.

 

 

signal-2024-05-26-13-42-26-311.jpg

#5723
1 hour ago, immamac said:

Why do that when you can just replicate off site? DR site is exactly that. A site that can scale up in the event of DR to serve traffic, it still has to have a complete copy of the data there. The data hotel I was talking about setting up as a tertiary never to serve up anything place is what you are suggesting, which isn't a bad idea I just don't know if it fucking matters. 

The reality is it wouldn't have been worth flipping to DR then having to wait till night to take the downtime and switch back to main and have circular replication or an extra redirect for another day because DNS propagation is potato even with 60s TTL. I'd rather just take the downtime up front if it's only an hour or 2 than to spend 12-16 hours of my own time making sure shit goes well and is fine.

Posts are only one part of the whole thing that makes the site go. 

Check this out. I had to use a Dallas DB server that I replicate to just in case some shit happens in the rack to serve traffic for the day. This is what that looks like. Sustained 300+Mbps and averaging well into the 400+Mbps range. Just for the database. I think people fail to recognize how big the site is constantly.

 

 

signal-2024-05-26-13-42-26-311.jpg

Is that 300-500Mbps for just the DB server? Or is that the front door app server? 

Sorry for all the probing questions lol, I have a customer that puts a ton of work into their DR but almost never fails over because of DNS headaches as you describe. They're trying to decide if it's really worth the power and man hours of maintaining a copy in a DR location given they've never actually used it in the last decade lol. 

It's crazy that the persistence isn't maintained in the DB or ES layer, where you could more easily keep a synchronized copy in a cluster config. I don't know the ins and outs of the board software, but it seems surprisingly stateful and brittle in that a new instance of the forum couldn't just read/write to the same data layer 

#5724

Simple script could automate the DNS failover, then manually fail it back.  

I also think you should mirror almost everything to the DR site, but replace all embedded pics with Billy_fellating_a_Thujone_donger.gif.

 

In all seriousness, what is the size of Surly in its entirety on disk, and do you tier the data storage, or is it all on one storage device?

#5725
5 minutes ago, PvilleStang said:

Simple script could automate the DNS failover, then manually fail it back.  

I also think you should mirror almost everything to the DR site, but replace all embedded pics with Billy_fellating_a_Thujone_donger.gif.

 

In all seriousness, what is the size of Surly in its entirety on disk, and do you tier the data storage, or is it all on one storage device?

It's on an array and each web node has local nvme cache serving it up for the internal cdn (media.surlyhorns.com). So it's tiered for access with cache misses getting pulled from a large disk array with plenty of dram cache and also replicated in real time (block) and every 15m to DR on the media side. It's measured in high single digit TB, but I'd say the hot data is in the 200-300GB range. 

The databases run on all flash. 

The problem isn't the mechanism of fail over. That's all automated. It's DNS propagation paired with intentional one way replication for active/passive. I have active/active intra rack. Database failovers that are planned are sub 10ms and unplanned I have not fully automated because I'd rather take downtime than risk split brain which is a bitch to remediate. 

Uptime isn't a priority because we aren't pumping revenue every second on this site. Data is the priority. 

#5728
23 minutes ago, thunderlounge said:

Worked ok for me a few minutes ago. 

I think it's related to new banner ads that are appearing at the bottom.  Being Burnt Ends, I'm guessing you don't get those.  Perhaps it's time to jump back in.

  • 2 weeks later...
#5732
14 minutes ago, immamac said:

Every time or just sometimes? 

Just for the last 24 hours (at most, prob more like 12 hours). In previously made posts and my attempt to post. 

The Mike Leach thread has a lot of examples.

Edited by TheStoicPaisano

  • 2 weeks later...
#5737

Concur. It’s posting slow. 
 

Also noticing streams are getting wonky. 
 

For example, a new post in the LOL thread was in my feed, but showed the post right before the new one, and looked like a visited link instead of a new one.

So if there were 10 posts, marked read, along comes post #11. It triggers the feed, but shows post #10 instead of #11, and appears read instead of unread. 

#5738

We haven't even hired a new baseball coach and it's already broken

#5745
8 hours ago, immamac said:

yeah the secondary web node doubles as an elasticsearch node and the high traffic caused some memory issues that I resolved. Shouldn't be an ongoing issue and set up an alert to handle.


Am not! Now, anyway. 

  • 2 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...
Football ... Basketball ... Baseball ... Other Sports ... Futbol ... 🤫995🤫 ... Gambling ... Movies & TV ... Music ... Hobbies ... Lulz ... Food & Travel ... Daily Texan ... Business & Markets ... Cloak Room ... Help ... For Sale ... Board Discussion ... Advertise... Tailgate Donations

Configure browser push notifications

Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.