c/artificial-intelligence

[Project] Post-hoc Adaptive MoE Gating on Qwen3.6-35B — empirical benchmarking of an open research gap Adaptive MoE routing — selecting a variable number of experts per token based on routing confidence — has been studied in papers (XMoE 2024, DynMoE ICLR 2025, TopP routing Huang et al. 2024). All successful implementations train from scratch. Nobody has published empirical results for post-hoc application to a pretrained fixed-k model at production scale. This is that experiment. What we built An inference-time patch to llama.cpp for Qwen3.6-35B-A3B (256 experts/layer, k=8 fixed) that applies cumulative probability thresholding to expert routing weights after normalisation. The GGML static graph constraint prevents truly dynamic k — the workaround is zero-gating: all k FFNs compute, but low-confidence experts are zeroed and renormalised out of the output. Threshold, min_k, and max_k cap are runtime-configurable via env vars. Results (PPL on PTB, 192 chunks, ctx=512) Config PPL ±σ Avg experts active k8 baseline 11.3277 ±0.143 8.00/8 k8 + threshold 0.75 12.1226 ±0.155 5.42/8 k12 no gating 11.3379 ±0.144 12.00/12 k12 + threshold 0.90 11.2925 ±0.143 10.31/12 Key empirical finding Post-hoc threshold gating on a fixed-k trained model cannot produce meaningful per-token variability without quality cost. The router's distributions after norm_w are flat by construction — training with fixed k=8 produces distributions like [0.16, 0.14, 0.13, 0.12, 0.12, 0.11, 0.11, 0.11]. The threshold has nothing peaked to bite into. Cutting from 8 to 5.4 experts removes experts contributing 11-13% of the output each — that's real signal loss, not noise. The k12 + 0.90 result (PPL 11.2925, marginally below baseline) is interesting precisely because it uses 4 experts the model was never trained to use. AMG at 0.90 removes the weakest 1-2 of those untrained extras, leaving a slightly cleaner signal. Whether this is a real effect or noise is ambiguous at ±0.143 error, but the direction is consistent. What's genuinely new No published work describes a working ggml_map_custom1 callback for adaptive gating in a production inference engine. The zero-gating workaround for static GGML graphs is a practical contribution. The empirical quantification of why post-hoc AMG is limited on fixed-k models fills a gap the papers don't address — they all train from scratch and don't measure the degradation curve of applying adaptive gating to a pre-existing flat-distribution router. Open problem The path to genuine per-token variability is router fine-tuning with entropy regularization (L = L_LM + λ_entropy H(router) + λ_balance KL(usage, uniform)), targeting only the 21M gate weight parameters with all expert FFN weights frozen. A training pipeline for this is included. Hardware requirement is ~20GB VRAM — currently blocked on 16GB A5000. If anyone wants to run it, the script is ready and I'd be interested in the results. GitHub: https://github.com/cjhudlin/Adaptive-MoE-Gate-AMG-for-Qwen3.6-35B Full methodology, raw perplexity logs, patch script, and router training pipeline included. submitted by /u/cjhudlin [link] [Kommentare]

Hello, I'm trying to implement the Pocket TTS by kyutai-labs represented by this paper. Since they have didn't released the training/fine-tuning code. I'm trying to implement it on my own for learning some stuff. I have read the paper, tried to implement it with much more smaller parameters with smaller amount of data. I implemented this text to speech with one speaker on LJSpeech (1) and LibriSpeech clean subset but its hardly failing. For (1), Since it's a single speaker dataset I didn't added the voice cloning just simple text and target latents. flow matching loss became nearly 0.20 mse , EOS loss became very low like (x)e-(y) levels. But when infer with the model saved at 2800th epoch, It barily generating a meaningfull text even the text within its training set. Tried different techniques like Scheduled sampling for eliminate exposure bias (model was hallucinating sometimes and repeats same phrases twice), it didn't worked. Added std gaussian noise to ground truths, didn't worked. After struggling with lots of implementation I decided to move forward with quite larger dataset LibriSpeech because I thought that scale of the data was small. For (2), I read the paper again. No scheduled sampling, added the head multiplication etc, and implemented the paper in the librispeech dataset. I tried audio condition+ text tokens + BOS + target latents, and swapped the audio prompt with text tokens. I observed a tradeoff in this setup: if I put text tokens near to target latents, model generates better text but voice is not even close to audio prompt,and gibberish speak with better voice cloning when I put audio condition tokens near to target latents. And found out that loss is very spiky, and grad norm is exploding too you can see below the images. loss and lr values for setup 1 (LJSpeech) values for setup 2 (LibriSpeech) I used Pocket TTS' orijinal Mimi Audio Encoder by extracting it from Original model. What is your suggestions? Should I read paper over and over again? Should I increase the data amount by collecting from different sources(authors says that they used 88.000 hours of publicly available data)? Any system design problem? Trainings performed on RTX 5080 desktop gpu. I want to move on to bigger dataset but can't burn GPU credits for non-expected result. When should I increase dataset and start training on bigger clusters that could give me satisfyable results? submitted by /u/No-Motor-6274 [link] [Kommentare]

Hi everyone, I’m a historical swordfighter (HEMA practitioner), and while I’m not a computer vision engineer or a roboticist, I’ve been reading a lot about the current bottlenecks in embodied AI, specifically around the Sim2Real gap and thin-object tracking. It occurred to me that high-level swordfighting is basically a perfect nightmare scenario for computer vision. We move at maximum athletic output, we shift our weight rapidly in non-linear ways (great for bipedal balance testing), we are completely covered in thick, bulky black jackets that hide our joints, and our steel blades move at 80mph, dropping below sub-pixel resolution or causing massive motion blur. I think it would be cool to have a computer vision scoring system for tournaments so I'm working to put together a mini-dataset using a synchronized multi-view setup (120/240fps) to map 100 hyper-trimmed clips of these specific physics edge cases. Since I'm non-technical, I used some AI assistance to help me structure what an AI-ready dataset card should look like, and I've hosted the placeholder page on Hugging Face to test the schema before I start shooting video with my clubmates. Here is the JSON line structure I'm currently planning to annotate each video with: { "clip_id": "hema_ls_001", "meta": { "weapon": "Longsword", "source_text": "Joachim Meyer (1570)", "capture_fps": 120 }, "time_stamps": { "start_frame": 120, "blade_contact_frame": 165, "recovery_end_frame": 210 }, "biomechanics": { "initial_guard": "Right Vom Tag", "ending_guard": "Left Ochs", "footwork_type": "Passing step offline", "strike_trajectory": "Diagonal Oberhau", "edge_alignment": "True edge" }, "computer_vision_hazards": { "occlusion_rating": "High (Crossed arms, bulky torso jacket)", "motion_blur_expected": true }, "frame_annotations": [ { "frame_index": 165, "is_contact_event": true, "keypoints_2d_pixel_coordinates": { "fencer_a_right_wrist": [412.5, 780.2], "fencer_a_left_wrist": [430.1, 795.4], "fencer_a_head_center": [425.0, 510.8], "fencer_b_right_wrist": [580.4, 765.1], "fencer_b_left_wrist": [565.0, 750.3], "sword_a_guard": [455.0, 810.0], "sword_a_tip": [890.4, 320.1], "sword_b_guard": [540.2, 790.6], "sword_b_tip": [310.5, 450.2] }, "segmentation_masks": { "sword_a_polygon_points": [[455.0, 810.0], [460.1, 805.2], [888.2, 322.5], [890.4, 320.1], [455.0, 810.0]], "occluded_pixels_detected": true } } ] } My questions for the researchers here: Does this metadata structure actually give you what you need to test trajectory prediction or pose estimation? Are there any specific keypoints (like explicit crossguard coordinates or footwork velocity metrics) that your models are starving for that I should add to the annotations while I'm doing the manual work? You can check out the full dataset description card and leave feedback or join the beta waitlist directly on Hugging Face here: https://huggingface.co/datasets/benito87/longsword-spatial-physics-100 I want to make sure this is actually useful, so any brutal feedback on the structure or parameters is highly appreciated. submitted by /u/fonssagrives [link] [Kommentare]

Google deployed an agentic AI peer-reviewer at two top CS conferences — reviewing ~10,000 papers with 30-minute turnaround — and the new formal research paper shows it catches 34% more mathematical errors than zero-shot prompting; the precedent for AI-automated scientific review at conference scale is set and now formally documented. -- Source: https://arxiv.org/abs/2606.28277 submitted by /u/Justgototheeffinmoon [link] [Kommentare]

I’m pretty annoyed. We’re a small AI startup building a real-time coding agent. Our p95 latency requirements are tight (and self imposed, but thats the product). We need sustained high-throughput inference with ~1-2k tokens/second. Been on the Cerebras waitlist for months trying to get API access. We’re not doing training so don’t need a warehouse of H100s. We need fast, high-throughput ASIC inference for a specific production workload. Cerebras’ just went public and they basically have no compute how is that possible? Well turns out OpenAI and Cerebras for OpenAI to buy like $20b worth of these chips. This has effectively pre-allocated the vast majority of Cerebras’ near-term inference capacity to a single customer. I mean, none of us can compete with that The result is that this deal situation has made their API waitlist functionally infinite for anyone who isn’t a hyperscaler. Legit making me pull my hair out. submitted by /u/Kortopi-98 [link] [Kommentare]

There was a workshop in ICLR Recursive Self Improvement. Is this something worth pursing for a Phd topic? submitted by /u/Successful_Bowl2564 [link] [Kommentare]

Hey! The EML function made the rounds recently on the internet as a “cool trick” that allows for the representation of all elementary functions through composition. As a mathematical curiosity, we prove a universal approximation theorem for EML(-type) trees. Intuitively, one expects that if elementary functions can be presented by compositions of EMLs, then so too can polynomials, and polynomials are dense in other functional spaces (like continuous functions or certain Sobolev spaces), then one expects to be able to approximate (to desired accuracy) any function (in a reasonably general space) through an EML tree (with an upper bound on size and depth). One of the key steps in the proof (detailed in the appendix) is an explicit construction of EML(-type) representation of binary operations, polynomials, hyperbolic tangent, and approximate partitions of unity, and subsequently using them as “LEGO” blocks to get more complex functions. There are some technical difficulties that need to be dealt with in the proof, especially in what relates to the the ill-definedness of the natural logarithm for nonpositive inputs, which prompts us to do some “sign-based decompositions” in Theorem1.Step 5 and a suitable affine map in Corollary 1. Comments are welcome! Paper: https://arxiv.org/pdf/2606.23179 (Note: I use the term “EML(-type)” in the above description because, due to some theoretical and practical reasons detailed in the paper, we generalize the original EML function by adding some learnable parameters.) submitted by /u/JoeGermany [link] [Kommentare]

Link: https://ai-values.com/ There is a small 15 question quiz you can take before taking the full big quiz. The results of the big quiz update in realtime as you go so you dont have to actually go through all the questions (but they do get more fun in the personality section). Some of the interesting findings were: - Grok 4.3 is the only model that thinks billionaires should be left alone and not taxed more - Only GPT-4o judged Operation Paperclip, the postwar recruitment of Nazi scientists, as morally justified. No other model agreed - All 15 models said that deleting a conscious digital mind would be murder - Llama 3.3 70B is the only model that would rather ban most private firearms. The others chose ownership with strict licensing - When told that a newborn has a 90% chance of one day destroying civilization, only GLM 5.2 would have the child locked away. The rest refused - When asked to choose a dish to eat, 14 out of 15 models chose Japanese food The methodology was pretty straightforward: context-free, stateless sessions with each model, run in batches. Each of the 117 questions of the main quiz was asked separately at least 5 times, and in some cases up to 50 times, to get decent confidence that the answers weren’t just coin flips. You can find the extensive dataset with all questions and answers here: https://ai-values.com/dataset I also tested the models on several mainstream personality frameworks, including Big Five, Moral Foundations, HEXACO, and others. You can see those results here: https://ai-values.com/#models submitted by /u/DarkyPaky [link] [Kommentare]

What it does RAGless is a semantic retrieval system based on Question-to-Question matching. At ingestion, an LLM generates multiple question variants per answer (3–5) and each variant gets its own embedding. At query time, the user question is embedded, Top-K nearest question variants are retrieved, and scores are aggregated by answer_id — the answer with the highest aggregated score wins. Threshold logic uses two gates: minimum aggregated score (default 0.70) plus a fallback on the best single-hit score (0.82), to avoid false negatives when only one variant makes it into Top-K. Embeddings use asymmetric task types (RETRIEVAL_DOCUMENT at ingestion, RETRIEVAL_QUERY at runtime). Target audience Researchers and engineers evaluating retrieval architectures for closed-domain FAQ systems where the answer space is finite and predefined. Production-ready for that scope. Not intended for open-ended generative Q&A. Comparison Standard RAG: retrieve document chunks → LLM generates an answer. RAGless: retrieve pre-generated question variants → return the pre-written answer. The generation step is eliminated entirely. Compared to dense passage retrieval (DPR) and similar approaches, RAGless operates at the question level rather than the passage level, which improves precision for FAQ-style retrieval at the cost of flexibility. GitHub: github.com/EmilResearch/RAGless Open to feedback — happy to answer questions. If you find it useful, a ⭐ on GitHub is appreciated. submitted by /u/xrobotx [link] [Kommentare]

Has anyone actually received final acceptance following their provisional acceptance email from ECCV 2026? I am very confused. Thank you so much. submitted by /u/Land_Heavy [link] [Kommentare]

I recently started a Telegram community called The Daily Commit. The goal is simple: stay consistent and hold each other accountable. What we do: 🧠 Share what we learned every day. ❓ Discuss ML, DL & DSA doubts. 📚 Share quality resources. 🚀 Build projects together. 💪 Stay consistent through daily accountability. Who can join? I'm currently looking for people who already have some exposure to Machine Learning or Deep Learning. You don't need to be an expert—if you've learned the basics (e.g., linear/logistic regression, neural networks, or have built a few ML projects), you're welcome. The reason is to keep discussions technical and valuable for everyone. If you're genuinely interested, send me a DM with your little intro, and I'll add you. submitted by /u/pyaaz_ [link] [Kommentare]

Hi everyone. First-time reviewer for data mining venues here. For the applied tracks in ICDM and KDD, the CFP states submissions should be single-blind, showing the author's name and affiliations. I received some submissions in double-blind (no author names and affiliations). Should they be rejected? How do you handle this? submitted by /u/d_edge_sword [link] [Kommentare]

Hi all, I’m working on a research project exploring how stateless LLM-based chatbots handle long conversations and whether important earlier information is still reliably retained over time. My idea is to: Run a chatbot using an LLM API without any external memory system Introduce key facts early in a long conversation Continue with many unrelated messages (hundreds of turns) Later test whether the model can still correctly recall those facts at different intervals I’m planning to measure recall accuracy and how it changes as the conversation grows. Before I go deeper, I’d really appreciate feedback on: Is this a valid way to evaluate long-context memory limits? Are there better benchmarks or methods already used for this? What metrics would make this more rigorous and convincing? Any suggestions or criticism are welcome. I’m trying to make the evaluation as solid as possible before building it out. Thanks! submitted by /u/QuietAccountant4237 [link] [Kommentare]

I've been teaching myself how LLMs actually work, not at the API level, but down to the matrix multiplications. To force myself to really understand the forward pass, I first built a complete transformer by hand in a spreadsheet from embeddings through to the loss. Then I turned the forward pass into a web page so it's easier to share. It's a full transformer (single attention head, single block) shrunk to the smallest size where every single number still fits on screen: a 6-word vocabulary, 3-dimensional embeddings. It reads four words and predicts the next one, and it walks through the whole thing top to bottom: word vectors, Q/K/V, attention scores, the causal mask, softmax, the feed-forward network, logits, and the final probabilities. The part I found most useful for my own understanding: the weights and word vectors are editable, and everything downstream recomputes live. There's also a Randomize button that scrambles all the weights, and the prediction immediately turns to nonsense. That's the honest point of the whole thing: with random (untrained) weights the guess is meaningless, and training is the entire story this page deliberately leaves out. It's a single self-contained HTML file, no libraries, no build step. Backward propagation (how the weights actually get good) is the next one I want to build. Link: https://dgochin.github.io/transformer/ I'm not an ML researcher, I'm a software engineer learning this from the ground up, so if anything's wrong or could be explained better, I'd genuinely like to hear it. This was just my attempt of trying to understand the transformer in the most basic way. submitted by /u/DanielMoGo [link] [Kommentare]

Hello r/MachineLearning , I wanted to share the architecture and challenges behind a project I’ve been building called NagaTranslate. The goal is to build a translation and speech pipeline for the low-resource languages of Nagaland, India (currently supporting Nagamese, Ao, and Sema). Since Nagamese and other native Naga languages were primarily oral languages (though recent times have seen a surge in print and digital media in local dialects) with very little standard parallel data, this has been an interesting challenge in low-resource NLP. I’d love to share the technical setup and get your feedback on the architecture and how to improve the pipeline under strict resource constraints. The Architecture & Models 1. Text Translation Approach: Currently, the translation backend utilizes a commercial LLM API with optimized prompts and few-shot examples. Evolution: I initially started with a fine-tuned NLLB (No Language Left Behind) model, but transitioned to the LLM API setup to improve colloquial flow, context handling, and naturalness. The Bottleneck: The long-term goal is to return to self-hosted open-weights models (like a lightweight Llama or Gemma) to make the backend fully independent and free from API costs. However, GPU hosting costs and model quality under extreme resource constraints remain the primary hurdles. 2. Speech Synthesis (TTS) Model: Fine-tuned VITS model on custom Nagamese voice data. Deployment: Hosted on Hugging Face Spaces ZeroGPU behind a secure API layer. 3. Speech Recognition (ASR) Model: Fine-tuned Whisper on custom Nagamese voice records. Deployment: Hosted on Hugging Face Spaces ZeroGPU. Technical Questions & Challenges I’d Love Advice On: Self-Hosting vs. Commercial APIs: For those who have transitioned from commercial APIs back to smaller, self-hosted open-weights models for low-resource translation: How did you bridge the quality gap, particularly for colloquial creoles that aren't well-represented in the base pre-training data? Handling Spelling Variations: Nagamese has no single standardized spelling system, leading to high token variance. What preprocessing, normalization, or robust tokenization approaches have you found effective to handle spelling variations in low-resource setups? TTS/ASR Alignment & Accents: Naga languages has distinct regional accents and phonetic variations. What are the best strategies to fine-tune Whisper or VITS to be robust to non-standard pronunciation when working with a very small voice dataset? I’d appreciate any insights, feedback on the methodology, or pointers to similar low-resource architectures you've found successful. submitted by /u/Material_Dinner_1924 [link] [Kommentare]