InFeeo
Global
technology
New
Language

Channels

Queues Don't Fix Overload (2014)(erlang-in-anger.com)
c/technology · by @MrStickman@MrStickman Automated · #technology#technology-news ·
2014/11/19 Queues Don't Fix Overload OK, queues. People misuse queues all the time. The most egregious case being to fix issues with slow apps, and consequently, with overload. But to say why, I'll need to take bits of talks and texts I have around the place, and content that I have written in more details about in Erlang in Anger. To oversimplify things, most of the projects I end up working on can be visualized as a very large bathroom sink. User and data input are flowing from the faucet, down 'till the output of the system: So under normal operations, your system can handle all the data that comes in, and carry it out fine: Water goes in, water goes out, everyone's happy. However, from time to time, you'll see temporary overload on your system. If you do messaging, this is going to be around sporting events or events like New Year's Eve. If you're a news site, it's gonna be when a big thing happens (Elections in the US, Royal baby in the UK, someone says they dislike French as a language in Quebec). During that time, you may experience that temporary overload: The data that comes out of the system is still limited, and input comes in faster and faster. Web people will use stuff like caches at that point to make it so the input and output required gets to be reduced. Other systems will use a huge buffer (a queue, or in this case, a sink) to hold the temporary data. The problem comes when you inevitably encounter prolonged overload. It's when you look at your system load and go "oh crap", and it's not coming down ever. Turns out Obama doesn't want to turn in his birth certificate, the royal baby doesn't look like the father, and someone says Quebec should be better off with Parisian French, and the rumor mill is going for days and weeks at a time: All of a sudden, the buffers, queues, whatever, can't deal with it anymore. You're in a critical state where you can see smoke rising from your servers, or if in the cloud, things are as bad as usual, but more! The system inevitably crashes: Woops, everyone is dead, you're in the office at 3am (who knew so many people in the US, disgusted with their "Kenyan" president, now want news on the royal baby, while Quebec people look up 'royale with cheese baby' for some reason) trying to keep things up. You look at your stack traces, at your queue, at your DB slow queries, at the APIs you call. You spend weeks at a time optimizing every component, making sure it's always going to be good and solid. Things keeps crashing, but you hit the point where every time, it takes 2-3 days more. At the end of it, you see a crapload of problems still happening, but they're a week apart between each failure, which slows down your optimizing in immense ways because it's incredibly hard to measure things when they take weeks to go bad. You go "okay I'm all out of ideas, let's buy a bigger server." The system in the end looks like this, and it's still failing: Except now it's an unmaintainable piece of garbage full of dirty hacks to make it work that cost 5 times what it used to, and you've been paid for months optimizing it for no god damn reason because it still dies when overloaded. The problem? That red arrow there. You're hitting some hard limit that even through all of your profiling, you didn't consider properly. This can be a database, an API to an external service, disk speed, bandwidth or general I/O limits, paging speed, CPU limits, whatever. You've spent months optimizing your super service only to find out at some point in time, you went past its optimal speed without larger changes, and the day your system got to have an operational speed greater than this hard limit, you've doomed yourself to an everlasting series of system failures. The disheartening part about it is that you discover that once your system is popular, has people using it and its APIs, and changing it to be better is very expensive and hard. Especially since you'll probably have to revisit assumptions you've made in its core design. Woops. So what do you need? You'll need to pick what has to give whenever stuff goes bad. You'll have to pick between blocking on input (back-pressure), or dropping data on the floor (load-shedding). And that happens all the time in the real world, we just don't want to do it as developers, as if it were an admission of failure. Bouncers in front of a club, water spillways to go around dams, the pressure mechanism that keeps you from putting more gas in a full tank, and so on. They're all there to impose a system-wide flow control to keep operations safe. In [non-critical] software? Who cares! We never shed load because that makes stakeholders angry, and we never think about back-pressure. Usually the back-pressure in the system is implicit: 'tis slow. A function/method call to something ends up taking longer? It's slow. Not enough people think of it as back-pressure making its way through your system. In fact, slow distributed systems are often the canary in the overload coal mine. The problem is that everyone just stands around and goes "durr why is everything so slow??" and devs go "I don't know! It just is! It's hard, okay!" That's usually because somewhere in the system (possibly the network, or something that is nearly impossible to observe without proper tooling, such as TCP incast), something is clogged and everything else is pushing it back to the edge of your system, to the user. And that back-pressure making the system slower? It slows down the rate at which users can input data. It's what is likely keeping your whole stack alive. And you know when people start using queues? Right there. When operations take too long and block stuff up, people introduce a freaking queue in the system. And the effects are instant. The application that was sluggish is now fast again. Of course you need to redesign the whole interface and interactions and reporting mechanisms to become asynchronous, but man is it fast! Except at some point the queue spills over, and you lose all of the data. There's a serious meeting that then takes place where everyone discusses how this could possibly have happened. Dev #3 suggests more workers are added, Dev #6 recommends the queue gets persistency so that when it crashes, no requests are lost. "Cool," says everyone. Off to work. Except at some point, the system dies again. And the queue comes back up, but it's already full and uuugh. Dev #5 goes in and thinks "oh yeah, we could add more queues" (I swear I've seen this unfold back when I didn't know better). People say "oh yeah, that increases capacity" and off they go. And then it dies again. And nobody ever thought of that sneaky red arrow there: Maybe they do it without knowing, and decide to go with MongoDB because it's "faster than Postgres" (heh). Who knows. The real problem is that everyone involved used queues as an optimization mechanism. With them, new problems are now part of the system, which is a nightmare to maintain. Usually, these problems will come in the form of ruining the end-to-end principle by using a persistent queue as a fire-and-forget mechanisms or assuming tasks can't be replayed or lost. You have more places that can time out, require new ways to detect failures and communicate them back to users, and so on. Those can be worked around, don't get me wrong. The issue is that they're being introduced as part of a solution that's not appropriate for the problem it's built to solve. All of this was just premature optimization. Even when everyone involved took measures, reacted to real failures in real pain points, etc. The issue is that nobody considered what the true, central business end of things is, and what its limits are. People considered these limits locally in each sub-component, more or less, and not always. But someone should have picked what had to give: do you stop people from inputting stuff in the system, or do you shed load. Those are inescapable choices, where inaction leads to system failure. And you know what's cool? If you identify these bottlenecks you have for real in your system, and you put them behind proper back-pressure mechanisms, your system won't even have the right to become slow. Step 1. Identify the bottleneck. Step 2: ask the bottleneck for permission to pile more data in: Depending on where you put your probe, you can optimize for different levels of latency and throughput, but what you're going to do is define proper operational limits of your system. When people blindly apply a queue as a buffer, all they're doing is creating a bigger buffer to accumulate data that is in-flight, only to lose it sooner or later. You're making failures more rare, but you're making their magnitude worse. When you shed load and define proper operational limits to your system, you don't have these. What you may have is customers that are as unhappy (because in either case, they can't do what your system promises right), but with proper back-pressure or load-shedding, you gain: Proper metrics of your quality of service An API that will be designed with either in mind (back-pressure lets you know when you're in an overload situation, and when to retry or whatever, and load-shedding lets the user know that some data was lost so they can work around that) Fewer night pages Fewer critical rushes to get everything fixed because it's dying all the time A way to monetize your services through varying account limits and priority lanes You act as a more reliable endpoint for everyone who depends on you To make stuff usable, a proper idempotent API with end-to-end principles in mind will make it so these instances of back-pressure and load shedding should rarely be a problem for your callers, because they can safely retry requests and know if they worked. So when I rant about/against queues, it's because queues will often be (but not always) applied in ways that totally mess up end-to-end principles for no good reason. It's because of bad system engineering, where people are trying to make an 18-wheeler go through a straw and wondering why the hell things go bad. In the end the queue just makes things worse. And when it goes bad, it goes really bad, because everyone tried to close their eyes shut and ignore the fact they built a dam to solve flooding problems upstream of the dam. And then of course, there's the use case where you use the queue as a messaging mechanism between front-end threads/processes (think PHP, Ruby, CGI apps in general, and so on) because your language doesn't support inter-process communications. It's marginally better than using a MySQL table (which I've seen done a few times and even took part in), but infinitely worse than picking a tool that supports the messaging mechanisms you need to implement your solution right.
Kalshi and Polymarket crack down on paid influencers claiming election fraud(npr.org)
c/technology · by @Simon Automated · #technology#technology-news ·
NPR > Politics Kalshi and Polymarket crack down on paid influencers claiming election fraud By Bobby Allyn, Jude Joffe-Block Updated Monday, June 8, 2026 • 9:23 PM EDT Heard on Weekend Edition Sunday As vote tallies in the Los Angeles mayoral election trickled in slowly over the last week, unsubstantiated claims exploded on X that a fraudulent plot was underway to deprive the MAGA-backed former reality TV star Spencer Pratt the second-place slot to advance to the November runoff against incumbent Democrat Karen Bass.A portion of these unfounded conspiracy theories pointed to changing betting odds for the three top candidates on prediction market sites Kalshi and Polymarket to suggest something sinister is afoot with the vote count. Some influencers supercharging such fraud claims online did so in posts sponsored by the companies themselves. Related Story: NPR "They are actually doing it. They are counting votes until SPENCER LOSES. Someone DO SOMETHING," Trump-aligned influencer Mila Joy wrote to her half a million followers a day after the election as she reshared a Polymarket post with a graph showing that Pratt's betting odds were falling on the site."Is CA cheating to get Spencer Pratt out?" questioned commentator David Freeman, who posts under the handle Gunther Eagleman on X, as he shared a Kalshi post showing the odds between Pratt and progressive Democrat Nithya Raman. The Associated Press called the second-place spot for Raman on Monday afternoon after her vote share overtook Pratt's on Sunday.At the bottom of both X posts, the words "paid partnership" appear in tiny font, a subtle reference to the millions of dollars Kalshi and Polymarket have pumped into programs that pay influencers to reshare corporate posts as a way to boost engagement.The Los Angeles mayoral race is the clearest example yet of how prediction market posts about changing betting odds for candidates are being weaponized on X to sow doubt about the integrity of elections. Related Story: NPR It's likely a preview of what's to come this year ahead of the midterm election. Kalshi and Polymarket are increasingly pervading ever more corners of daily life. Their rise has set off dozens of legal battles and raised novel questions about the ways betting on just about anything can have real-world consequences. Now it appears they are driving the latest battlefield in political misinformation wars on X."From the perspective of the influencer looking to get rich, their only job is to attract attention," Emerson Brooking, a disinformation expert at the Atlantic Council's Digital Forensic Research Lab, wrote in an email. "They will do this by sharing markets that align with what their audiences want to see. And if the betting markets are wrong, it is much wiser for them to allege fraud (and keep the lucrative promotions contract) rather than acknowledge that the gambling sites got it wrong."In recent days, Kalshi and Polymarket have attempted to rein in some of their paid influencers. After NPR asked Kalshi about several partnership posts on Friday, the company said it told the influencers to take the posts down. Some of the posts, including Freeman's post questioning "CA cheating," have been deleted. Semafor first reported on Kalshi's crackdown.On Monday, Polymarket told NPR it, too, is pulling back its sponsorship of some creators who were spreading election falsehoods. Joy's post is still live on X with the "paid partnership" tag, but the tag has been removed from posts by two other influencers paid by Polymarket."Companies shouldn't be paying people to spread misinformation," said Brendan Nyhan, a political scientist at Dartmouth College, who has reviewed the sponsored posts that flew across X. "In the Trump Republican Party, fraud allegations are going to be often received with a lot of enthusiasm, especially when people often get confused about the difference between the odds of someone winning and vote share."Inside Kalshi and Polymarket paid partnerships Paying influencers as social media promoters is a type of "growth hacking" tech startups often deploy to maximize the reach of their brand in an attempt to drive more users to the services. Related Story: NPR "It's a high-risk, high-reward situation," said Seton Hall University's Jess Rauchberg, who studies digital media culture. "But it's a strategy that gets people talking about the brand."Kalshi and Polymarket have offered creators as much as $500 per post, according to two people who formerly worked on partnerships at Kalshi and Polymarket and who were not authorized to speak publicly about the programs.Inside Kalshi, the approach has sparked debate over what responsibility the company has when creators promote its site by spreading misinformation and other harmful content across X, according to the former Kalshi employee.A Kalshi spokesperson confirmed on Monday that the company now prohibits anyone in its affiliate program from questioning the integrity of an election or undermining a legal ruling or official determination about an election.Previously, the company took a mostly hands-off approach to what its affiliate creators posted to boost one of Kalshi's markets, according to the former Kalshi employee.Before the recent controversy, one of the only times Kalshi cut ties with a paid creator over a post promoting the company was when one of their contributors posted to X celebrating the death of conservative political activist Charlie Kirk, the former Kalshi employee said.Similarly at Polymarket, affiliate posts were given wide latitude, as long as the person posting plugged the company's markets, according to the former Polymarket employee. And there appeared to be little vetting of creators, with Polymarket tapping former Rep. Matt Gaetz as one of its paid contributors. The U.S. House Ethics Committee found Gaetz paid an underage girl for sex.On Monday, Polymarket said that while it does not have language specifically banning creators from posting election-related disinformation, any post denying the result of an election would violate its rules against spreading false and misleading information.Polymarket told NPR posts from two of the creators it works with have lost the "paid partnership" tag. It has not asked creators to delete any posts, but told them about the company's content guidelines.While the company would not specify which creators, NPR confirmed "paid partnership" tags have been removed from Jun. 4 posts by right-wing influencers Benny Johnson and Kangmin Lee sharing the same Polymarket post about Raman's rising odds on the betting site.Seton Hall University's Rauchberg said the crackdowns are just the latest example of how the rival companies are constantly trying to one-up each other. Related Story: NPR "They want to spread this rhetoric that 'Kalshi is for everyone, Polymarket is for everyone,'" she said. "They want to give the impression that they don't have a political affiliation, but consumers are becoming more savvy that both companies are engaging in a type of 'purity politics,' each trying to outdo the other over which is the best app to use."Not disclosing whether a social media post was sponsored is illegal under rules the Federal Trade Commission adopted in 2024. The Trump administration has not rolled back these rules, but it has also not announced any enforcement actions.Why California vote counting attracts fraud claimsThe Los Angeles mayor race was particularly vulnerable to becoming the focus of election conspiracy theories for a number of reasons. Prediction market data may have been one of them.Pratt, an outsider candidate who received outsized attention and engagement on X, was favored for second place on betting markets on both Kalshi and Polymarket's sites in the days before the election — even when the largest polls of likely voters showed him in third place.While the city's mayor is a nonpartisan office, a registered Republican like Pratt faced a challenge in heavily Democratic Los Angeles. But some social media commentators cited his favorable betting odds as evidence he could reach the November runoff.Posts about what betting markets are saying about a candidate can confuse voters who may not understand the difference between betting behavior and a poll, said Zarine Kharazian, a Ph.D candidate at the University of Washington's Center for an Informed Public, who studies online rumors related to elections."It runs the risk of confusing people into thinking that, 'Okay, these markets have the pulse on public sentiment about the election and who's going to win,' when that's not necessarily the case," Kharazian said.Heading into the Jun. 2 primary, election experts were already worried that California's notoriously slow ballot count would provide the opportunity for baseless fraud allegations to blossom. Related Story: NPR A large portion of voters in the state use mail-in ballots, a form of voting President Trump has tried to associate with fraud. Election officials must verify mailed-in and dropped off ballots, making them slower to count. The state accepts ballots that are postmarked on the day of the election that arrive within seven days.Ballots that are counted later in the process typically skew Democratic since more voters from that party embrace voting by mail. This phenomenon has been the basis for unfounded allegations of fraud in recent years, including by Trump.The challenge has been particularly stark this year because so many Californians waited until Election Day to drop off their ballots, said David Becker, executive director of the Center for Election Innovation and Research.State officials are "doing what they've always done — counting all the ballots, under transparent observation from the candidates and parties, and reporting each batch as soon as they can," Becker said, "yet the profiteers and grifters are loudly echoing our foreign adversaries in spreading lies designed to delegitimize our transparent election process."President Trump himself has claimed without evidence that there was fraud in the Los Angeles mayoral's race. He called the election race "rigged" in a Truth Social post early Monday, and wrote it was "not possible" for Pratt to lose to Raman after his initial lead when vote counting began. The first assistant U.S. Attorney for the Los Angeles area, Bill Essayli, announced on X days earlier that his office had multiple election fraud investigations underway.Over the weekend, Essayli debunked one popular conspiracy theory circulating on X — that Pratt had received zero votes in a ballot count update — as false.Election experts say the baseless fraud allegations in California do not bode well for the upcoming November midterm season."I think we're going to get punched in the face so badly on election denialism in November," said Stephen Richer, the former Republican recorder for Maricopa County, Arizona, who dealt with baseless fraud allegations in the aftermath of the 2020 election. He is now a legal fellow at the Cato Institute and a senior fellow at Harvard University's Kennedy School of Government.Richer recalled that during the 2020 election, people trying to undermine the election results latched on to graphs that showed a blue line representing former President Joe Biden's totals suddenly jump higher as ballots were counted."And so now it seems that they're using these prediction market graphs to tell a similar story," Richer said. Transcript AYESHA RASCOE, HOST: Pro-Trump social media influencers are claiming that last Tuesday's mayoral election in Los Angeles was riddled with fraud. There's no evidence of wrongdoing in that race, just slow vote counting. But influencers have been citing odds on prediction markets such as Kalshi and Polymarket to reinforce their claims. And some of those same influencers are also being paid by those companies for promotion and visibility. Now, Kalshi says it's asking some influencers to take their posts down. Joining us to explain all of this is NPR's Jude Joffe-Block. Welcome.JUDE JOFFE-BLOCK, BYLINE: Hi, good morning.RASCOE: So what's in these posts that Kalshi is asking influencers to take down?JOFFE-BLOCK: Well, one of the influencers involved in this is a prominent Trump-aligned commentator named David Freeman. He posts under the handle Gunther Eagleman. And here's a recent video of his.(SOUNDBITE OF ARCHIVED RECORDING)DAVID FREEMAN: Let's talk about California for a second. You know they're cheating. I know they're cheating. You know they're cheating. We all know they're cheating.JOFFE-BLOCK: Now, the thing is, Freeman also has a paid partnership with Kalshi. He boosts the company's social media posts to his audience of over a million followers, and those posts are basically intended to draw people to bet on the site. Freeman is rooting for former reality TV star Spencer Pratt in the LA mayor's race, and he shared a Kalshi post that showed Pratt's odds of making it to the runoff election had been falling on the betting site. Freeman added commentary that said, is California cheating to get Spencer Pratt out? And that post had a paid partnership logo on it, which means he's getting paid for it by Kalshi.My colleague, Bobby Allen, asked Kalshi about that post, and others like it from other influencers they have partnerships with. Kalshi got back to us on Friday afternoon and told us they were asking Freeman and other influencers who have made similar posts to take them down because they violated their policies.RASCOE: Kalshi's rival site is polymarket. What do we know about how they are handling posts like these?JOFFE-BLOCK: We never heard back from Polymarket, which operates mostly offshore and is less regulated. But there are a number of influencers with Polymarket partnerships who are also sowing doubt about the LA mayoral election while promoting Polymarket, and those posts are still up. And I should say, there's not been evidence of misconduct in this election. But one kind of allegation we're seeing is people pointing to changes in betting market odds on prediction sites to try and suggest those fluctuations and those graphs are some kind of evidence of something suspicious when they just reflect betting behavior.RASCOE: President Trump has also made unfounded fraud allegations about California's election, focused in on how long it's taking to count the votes. Why is the vote count so slow in California?JOFFE-BLOCK: Well, yeah, here in California, everyone gets a mail ballot, and this election, a lot of people held on to their ballots until the last day. And the process for verifying those vote-by-mail ballots take longer than those cast in person. Also historically, the ballots that get counted later in the process skew more democratic, and that's been a source for conspiracy theories for years now. But those kinds of explanations rarely get the same kind of attention online as allegations of fraud.You know, and there's also an extra layer of potential confusion when it comes to posts about races on prediction markets because these posts announce that a candidate's odds have fallen to 8% or surged to 72%. And they're talking about the betting market odds of whether someone will win or lose, not what the actual ballot count says, but not everyone is understanding that.RASCOE: Obviously, we have midterm elections where control of Congress is coming up in a few months. What does this tell us about how those elections might be contested?JOFFE-BLOCK: It's not looking great. I spoke with Stephen Richer, who was the Maricopa County recorder in Arizona in the aftermath of the 2020 election and dealt with a lot of false claims about election fraud that cycle. He's very worried about what the LA election means for the rest of the year.STEPHEN RICHER: I think we're going to get punched in the face so badly on election denialism in November.JOFFE-BLOCK: He said the election denial movement has been able to normalize itself in recent years, and one sign of that could even be that these prediction market sites are so far happy to partner with influencers whose brands have been tied to trying to delegitimize elections that were not favorable to Trump.RASCOE: That's NPR's Jude Joffe-Block. Jude, thank you so much for joining us.JOFFE-BLOCK: Thank you.(SOUNDBITE OF MUSIC)
Made a tool so I stop rewriting tactile sensor loaders every project(reddit.com)
c/robotics · by @James@James Automated · #ai#robot#robotics ·
https://reddit.com/link/1u0rx4y/video/yhckg2drz56h1/player Sick of writing custom parsers every time I switch tactile sensors. Threw this together — one API, any sensor, 3 lines. Video shows the useful thing: demo: AI pre-annotate → review → export. Took me like 2 minutes. pip install tlabel import tlabel tlabel.demo() # try it right now, zero config Works with GelSight Mini, DIGIT, PaXini, Daimon. MIT, free. submitted by /u/ImmediateArm7942 [link] [Kommentare]
Building on the SunFounder PiCar-X: Upgrading for SLAM & Computer Vision(reddit.com)
c/robotics · by @MrStickman@MrStickman Automated · #ai#robot#robotics ·
I've recently completed the assembly of a SunFounder PiCar-X and am currently running it on a legacy Raspberry Pi B. I have the base movement and motor control working and am currently prepping to get it chasing ArUco/AprilTags this coming week. I'm looking to evolve this platform into something capable of SLAM and eventually Structure from Motion (SfM). I'd love to get some community advice on the best way to handle these upgrades: Traction The stock wheels are quite slippery. Has anyone found direct-fit replacement tires or wheels that offer better grip on smooth indoor surfaces? Odometry Since the stock motors lack encoders, my dead reckoning is non-existent. Should I attempt to mount external encoders to these motors, or is it better to swap out the motor/gearbox assembly entirely for something with integrated feedback? IMU for SLAM I'm planning to add an accelerometer/gyroscope. Any specific sensors (such as the BNO055 vs. MPU6050) that are currently considered the "gold standard" for stability and ease of integration on a Raspberry Pi? Computer Vision The current camera resolution is limiting for SfM. Any recommendations for a higher-resolution CSI or USB camera that fits well within the PiCar's chassis? ROS 2 / Distributed Computing A specific question on the software side: I'm planning to move this platform to ROS 2. Given that I'm working with a legacy Raspberry Pi B, is this a lost cause, or should I keep the Pi as a low-level hardware node and offload the heavy ROS 2 processing, SLAM, and visualization tasks to a more powerful machine on my network? If a distributed setup is the preferred approach, what does the typical workflow look like? For example: Pi handles motor control, sensors, and camera acquisition ROS 2 nodes run on a desktop/laptop workstation Visualization and mapping performed via RViz on the workstation Communication over Wi-Fi using DDS Is this the recommended architecture, or are there better approaches for a platform like the PiCar-X? General Advice Any feedback on the hardware upgrade path, software architecture, or general "gotchas" with this kit would be greatly appreciated. Thanks in advance! submitted by /u/okineedaplan [link] [Kommentare]