Good day! I’m looking for is a way of creating rules to intercept, modify, drop, and replace HTTP requests and responses, hopefully even with regex(or similar) capabilities.
The best extension I’ve found that seems to suit those needs is Requestly. However, it seems like they have some shady practices of bought/bot reviews, like here on AlternativeTo.net, where you can see the review are made by accounts that are created the same day of the review, and never used since. The same pattern can be found on ProductHunt.
Is there perhaps an audit of their Github repo somewhere?
I’ve also looked at apps like mitmproxy, but I was hoping for a solution that is in-browser.
I know that Firefox and Chromium has the built-in dev tools for this, but this is only applied with the dev tools actively open; I’m looking for a more persistent solution.
Please let me know if this is not the place to ask, and if there are other places I should try and look instead/also.
Edit
My goal is to do something to the effect of uBlock Origin, but instead of just blocking/hiding, either replace with local files, or intercept req/res in order to manipulate them favorably, without being detected. I don’t know what uBlock does under the hood though, apart from its resource blocking and CSS-derived hiding.
Example: Watching a video on youtube, an ad is about to get loaded, but instead of the hiding/blocking strategy uBlock uses, intercept the GET/POST, save the important flags that are uniquely served to your device that would indicate that you have successfully been served the ad, drop the rest, and then answer with what would be a valid response for “I have watched the ad in its entirety”. So the server basically saying “Here, I give you this page and this script with both vital and ad contents. I now expect you to provide the corresponding hash that these two files will create through a series of functions. If you don’t, I will assume you’re blocking me, and I won’t provide further contents.”, and I’ll simply respond with “Here’s your hash! wink”.
Essentially, I wish to experiment with trying to be completely invisible in the blocking, by providing responses as if I have loaded and watched the ad, with all anti-adblock implementations through scripts and dynamic loading “intact” and unaware.
Another commenter mentions this as an xy problem.
I think it’s a reasonable point, and take it to mean that you might be jumping to a solution that is convoluted or unnecessary.
What is your actual goal: the problem you care about solving? It’s great you’ve thought about possible solutions, but including the end goal in the problem description will help domain experts share the best possible guidance.
I most likely am going a bit to the obscure/convoluted solution, yes.
My goal is to do something to the effect of uBlock Origin, but instead of just blocking/hiding, either replace with local files, or intercept req/res in order to manipulate them favorably, without being detected. I don’t know what uBlock does under the hood though, apart from its resource blocking and CSS-derived hiding.
Example:
Watching a video on youtube, an ad is about to get loaded, but instead of the hiding/blocking strategy uBlock uses, intercept the GET/POST, save the important flags that are uniquely served to your device that would indicate that you have successfully been served the ad, drop the rest, and then answer with what would be a valid response for “I have watched the ad in its entirety”.
So the server basically saying “Here, I give you this page and this script with both vital and ad contents. I now expect you to provide the corresponding hash that these two files will create through a series of functions. If you don’t, I will assume you’re blocking me, and I won’t provide further contents.”, and I’ll simply respond with “Here’s your hash! *wink*”.Essentially, I wish to experiment with trying to be completely invisible in the blocking, by providing responses as if I have loaded and watched the ad, with all anti-adblock implementations through scripts and dynamic loading “intact” and unaware.
You’re quite right, I should include my goal in the post - editing it in now.
Wow! That’s ambitious!
My intuition is that looking at something open source (like greasemonkey? Or yt-dlp?) would be the most productive way to start. In my experience, getting up to speed on a complex topic is a place where AI is really useful. Maybe pose this question to chat gpt and work from there?
Sorry I don’t have something more concrete to provide, but you have set your sights quite high. It’s an ongoing cat and mouse game with all of the existing tools, so even if you solve it today, it will require ongoing attention to keep working.
Good luck!
How about:
- uBlock Origin: To block parts of a web-page
- GreaseMonkey / TamperMonkey / ViolentMonkey: To alter request/responses through JavaScript
Perhaps one of those can suite your needs
uBlock Origin I very much like, and I think it’s near-impossible for me to use a browser without it installed anymore. But as far as I know, it’s only used for blocking (dropping network requests) and hiding (CSS-like rules), but no manipulation?
The monkeys I’ve also used a little bit of, but I wasn’t aware that they supported being run ahead of the rest of the page like how I wanted, so this is definitely worth looking into, I appreciate it!
For reference, I found this answer on StackOverflow, which is promising.
ZAP may be overkill but also worth looking into if you haven’t already.
I haven’t heard of ZAP before, so this is new and intriguing! Thanks!
I’m wondering if this is a bit of a xy problem.
Because there’s tools like Fiddler and Hoppscotch that come to mind but it’s a bit outside your scope.
Yeah, a bit of an xy problem, I’ll admit. My ultimate goal would be invisible blocking/hiding from certain expected behaviors like the ones anti-adblockers employ. I’m not sure if what I have in mind specifically exists or not, but I find it fun to tweak and experiment with these kind of things. So I don’t necessarily have a problem I’d like to solve, it’s more an ask for directions/experiences.
I have tried Fiddler before, and it almost has the capabilities I’m looking for (it replaces an entire file; nothing granular), but it was a bit of a hassle to get working well last time I tried it, with the CA certificate, decrypting the TLS and stuff.
I’m wondering if you could do this with squid proxy. It’s been ages since I used it but I recall the ability to write PHP apps for it that “mangle” connections.
