Off-and-on trying out an account over at @tal@oleo.cafe due to scraping bots bogging down lemmy.today to the point of near-unusability.

  • 3 Posts
  • 992 Comments
Joined 2 years ago
cake
Cake day: October 4th, 2023

help-circle

  • tal@lemmy.todaytoSelfhosted@lemmy.worldRouter of choice?
    link
    fedilink
    English
    arrow-up
    1
    ·
    6 hours ago

    Many open source operating systems exist that can turn a computer with multiple NIC’s into a router

    Minor nitpick, but if you’re planning on sticking a NIC into a machine to make it a router, it’s probably more cost-effective to get a single NIC with multiple Ethernet ports than multiple NICs.


  • tal@lemmy.todaytoSelfhosted@lemmy.worldRouter of choice?
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    10 hours ago

    Looking at using older hardware we have spare (a MacBook Pro 2012 or rpi4) seem to have a track record of underperforming

    In what sense?

    I’m having trouble finding good options particularly in regards to openwrt at least.

    Everything I can get in local stores isn’t supported by openwrt (neither are the current routers).

    IIRC, OpenWRT tends to support older hardware. I once bought new hardware to run it, so I know that it’s been out there, but if you want something to run OpenWRT and aren’t too fussed about having the latest hardware, you can probably grab something off eBay or something, especially if what you care about isn’t the WiFi side of things, where things have changed over time. Might be possible to run a USB WiFi adapter or something, if you want the latest WiFi protocol.

    Would the MacBook Pro or rpi4 with a second Ethernet nic running a firewall before the routers also fix the issue of not getting security updates?

    Pretty much, if you’re talking Internet-facing stuff. I mean, you might still want updates for, I dunno, NTP updates or something where the router talks to the Internet. And if it’s doing WiFi and there’s some vulnerability associated with that, theoretically you could be attacked locally. In general, I wouldn’t worry too much. There are probably a ton of unsupported, unupdated Internet of Things devices on LANs all over the place, so shrugs. It’d be nice to have maintenance and security updates for everything, but in practice, there’s probably a lot of stuff that is always going to be unmaintained on most LANs. Smart TVs, printers, whatever. Maybe we should change that, but as things stand, kinda the norm.



  • The NEMA 1-15 power cable becomes the most important item in the room when people discover their specialized devices cannot recharge without it. It is frankly hilarious how much we obsess over liquid cooling and fiber optics while our daily sanity relies on a design that has barely changed since the dawn of the lightbulb.

    Only peripherally-related, but on that note:

    • The most-common lightbulb socket in the US (the “Edison screw”) dates back to 1909.

    • The 1/8-inch headphones jack isn’t identical to the original standard, but it’s a direct decendant (and electrically-compatible with) the 1/4 inch mono phone jack from (checks) 1877. Since that time, we’ve had variants with more rings (stereo, microphone, balanced audio) added and the size reduction to 1/8-inch for smaller devices. But the basic standard has been around for quite a while.


  • He could probably run an NFS server that isn’t a closed box, and have that just use the Synology box as storage for that server. That’d give whatever options Linux and/or the NFS server you want to run have for giving fair prioritization to writes, or increasing cache size (like, say he has bursty load and blows through the cache on the Synology NAS, but a Linux NFS server with more write cache available could potentially just slurp up writes quickly and then more-slowly hand them off to the NAS).

    Honestly, though, I think that a preferable option, if one doesn’t want to mess with client global VM options (which wouldn’t be my first choice, but it sounds like OP is okay with it) is just to crank up the timeout options on the NFS clients, as I mention in my other comment, if he just doesn’t want timeout errors to percolate up and doesn’t mind the NAS taking a while to finish whatever it’s doing in some situations. It’s possible that he tried that, but I didn’t see it in his post.

    NFSv4 has leases, and — I haven’t tested it, but it’s plausible to me from a protocol standpoint — it might be possible that it can be set up such that as long as a lease can be renewed, it doesn’t time out outstanding file operations, even if they’re taking a long time. The Synology NAS might be able to avoid taking too long to renew leases and causing clients to time out on that as long as it’s reachable, even if it’s doing a lot of writing. That’d still let you know if you had your NFS server wedge or lost connectivity to it, because your leases would go away within a bounded amount of time, but might not time out on time to complete other operations. No guarantee, just it’s something that I might go look into if I were hitting this myself.


  • That’s a global VM setting, which is also going to affect your other filesystems mounted by that Linux system, which may or may not be a concern.

    If that is an issue, you might also consider — I’m not testing these, but would expect that it should work:

    • Passing the sync mount option on the client for the NFS mount. That will use no writeback caching for that filesystem, which may impact performance more than you want.

    • Increasing the NFS mount options on the client for timeo= or retrans=. These will avoid having the client time out and decide that the NFS server is taking excessively long (though an operation may still take longer to complete if the NFS server is taking a while to respond).


  • Game streaming serices are never going to catch on because the capital needed to build out the infrastructure is ridiculous.

    I don’t know about “never”, but I’ve made similar arguments on here predicated on the cost of building out the bandwidth — I don’t think that we’re likely going to get to the point any time soon where computers living in datacenters are a general-purpose replacement for non-mobile gaming, just because of the cost of building out the bandwidth from datacenter to monitor. Any benefit from having a remote GPU just doesn’t compare terribly well with the cost of having to effectively have a monitor-computer cable for every computer that might be used concurrently to the nearest datacenter.

    But…I can think of specific cases where they’re competitive.

    First, where power is your relevant constraint. If you’re using something like a cell phone or other battery-powered device, it’s a way to deal with power limitations. I mean, if you’re using even something like a laptop without wall power, you probably don’t have more than 100 Wh of battery power, absent USB-C and an external powerstation or something, due to airline restrictions on laptop battery size. If you want to be able to play a game for, say, 3 hours, then your power budget (not just for the GPU, but for everything) is something like 30W. You’re not going to beat that limit unless the restrictions on battery size go away (which…maybe they will, as I understand that there are some more-fire-safe battery chemistries out there).

    And cell phone battery restrictions are typically even harder, like, 20 Wh. That means that for three hours of gaming, your power budget because of size constraints on the phone is maybe about 6 watts.

    If you want power-intensive rendering on those platforms doing remote rendering is your only real option then.

    Second, there are (and could be more) video game genres where you need dynamically-generated images, but where latency isn’t really a constraint. Like, a first-person shooter has some real latency constraints. You need to get a frame back in a tightly bounded amount of time, and you have constraints on how many frames per second you need. But if you were dynamically-rendering images for, I don’t know, an otherwise-text-based adventure game, then the acceptable time required to get a new frame illustrating a given scene might expand to seconds. That drastically slashes the bandwidth required.

    What I don’t think is going to happen in the near future is “gaming PC/non-portable video game consoles get moved to the datacenter”.




  • What makes this worse is that git servers are the most pathologically vulnerable to the onslaught of doom from modern internet scrapers because remember, they click on every link on every page.

    The especially disappointing thing is that, for the specific case that Xe was running into, a better-written scraper could just recognize that this is a public git repository and just git clone the thing and get all the useful code without the overhead. Like, it’s not even “this scraper is scraping data that I don’t want it to have”, but “this scraper is too dumb to just scrape the thing efficiently and is blowing both the scraper’s resources and the server’s resources downloading innumerable redundant copies of the data”.

    It’s probably just as well, since the protection is relevant for other websites, and he probably wouldn’t have done it if he hadn’t been getting his git repo hammered, but…

    EDIT: Plus, I bet that the scraper was requesting a ton of files at once from the server, since he said that it was unusable. Like, you have a zillion servers to parallelize requests over. You could write a scraper that requested one file at once per server, which is common courtesy, and you’re still going to be bandwidth constrained if you’re schlorping up the whole Internet. Xe probably wouldn’t have even noticed.


  • https://en.wikipedia.org/wiki/National_Helium_Reserve

    The National Helium Reserve, also known as the Federal Helium Reserve, was a strategic reserve of the United States, which once held over 1 billion cubic meters (about 170,000,000 kg)[a] of helium gas.

    The Bureau of Land Management (BLM) transferred the reserve to the General Services Administration (GSA) as surplus property, but a 2022 auction[10] failed to finalize a sale.[11] On June 22, 2023, the GSA announced a new auction of the facilities and remaining helium.[12] The auction of the last helium assets was due to take place in November, 2023.[13] Though the last of the Cliffside reserve was to be sold by November 2023, more natural gas was discovered at the site than was previously known, and the Bureau of Land Management extended the auction to January 25, 2024 to allow for increased bids.[14] In 2024 the remaining reserve was sold to the highest bidder, Messer Group.[15]

    Arguably not the best timing on that.


  • tal@lemmy.todaytoSelfhosted@lemmy.worldYour logging is probably down
    link
    fedilink
    English
    arrow-up
    3
    ·
    edit-2
    12 days ago

    Sure. What that guy is using is actually not the most-interesting diagram style, IMHO, for automatic layout of network maps, if you want large-scale stuff, which is where the automatic layout gets more interesting. I have some scripts floating around somewhere that will generate very large network maps — run a bunch of traceroutes, geolocate IPs, dump the results into an sqlite database, and then generate an automatically laid-out Internet network map. I don’t want to go to the trouble of anonymizing the addresses and locations right now, but if you have a graphviz graph and want to try playing with it, I used:

    goes looking

    Ugh, it’s Python 2, a decade-and-a-half old, and never got ported to Python 3. Lemme gin up an example for the non-hierarchical graphviz stuff:

    graph.dot:

    graph foo {
        a--b
        a--d
        b--c
        d--e
        c--e
        e--f
        b--d
    }
    

    Processed with:

    $ sfdp -Goverlap=prism -Gsep=+5 -Gesep=+4 -Gremincross -Gpack -Gsplines=true -Tpdf -o graph.pdf graph.dot
    

    Generates something like this:

    That’ll take a ton of graphviz edges and nicely lay them out while trying to avoid crossing edges and stuff, in a non-hierarchical map. Get more complicated maps that it can’t use direct lines on, it’ll use splines to curve lines around nodes. You can create massive network maps like this. Note that I was last looking at graphviz’s automated layout stuff about 15 years ago, so it’s possible that they have better layout algorithms now, but this can deal with enormous numbers of nodes and will do reasonable things with them.

    I just grabbed his example because it was the first graphviz network map example that came up on a Web search.






  • You have all your devices attached to a console server with a serial port console set up on the serial port, and if they support accessing the BIOS via a serial console, that enabled so that you can access that remotely, right? Either a dedicated hardware console server, or some server on your network with a multiport serial card or a USB to multiport serial adapter or something like that, right? So that if networking fails on one of those other devices, you can fire up minicom or similar on the serial console server and get into the device and fix whatever’s broken?

    Oh, you don’t. Well, that’s probably okay. I mean, you probably won’t lose networking on those devices.


  • You have remote power management set up for the systems in your homelab, right? A server set up that you can reach to power-cycle other servers, so that if they wedge in some unusable state and you can’t be physically there, you can still reboot them? A managed/smart PDU or something like that? Something like one of these guys?

    Oh. You don’t. Well, that’s probably okay. I mean, nothing will probably go wrong and render a device in need of being forcibly rebooted when you’re physically away from home.


  • You have squid or some other forward http proxy set up to share a cache among all the devices on your network set up to access the Web, to minimize duplicate traffic?

    And you have a shared caching DNS server set up locally, something like BIND?

    Oh. You don’t. Well, that’s probably okay. I mean, it probably doesn’t matter that your devices are pulling duplicate copies of data down. Not everyone can have a network that minimizes latency and avoids inefficiency across devices.