AI交易“风向标”：Meta发债“一波三折”
华尔街见闻Meta选举日发债460亿美元"一波三折"：遭标普下调评级、微软抢占市场、政治变局三重夹击，CFO亲自路演500名投资者，最终付出真金白银——最长期限债券利差较4月扩大90个基点。民主党"蓝色浪潮"令AI数据中心监管风险骤升，科技债利差上周持续走阔，AI资本支出能否转化为回报成市场最大疑问。
Playing with the STM32MP2
Lobsters
What can I realistically do with local AI on a Mac mini M5 Pro (48GB)?
Lobsters
I just got a Mac mini with an M5 Pro (base CPU config) and 48GB of unified memory. I want to run as much AI locally as makes sense, but it's hard to separate what actually works day to day from benchmark hype.
What I'd like to do:
- Coding: autocomplete and agentic workflows with local models
- Image generation
- Anything else genuinely useful: transcription, RAG over my notes, etc.
Questions:
- Which models and quantizations fit comfortably in 48GB and are actually usable, not just "it runs"?
- Which runtime and tools: MLX, llama.cpp, Ollama, LM Studio, ComfyUI, something else?
- What's your setup, and what did you give up on and go back to cloud APIs for?
Real tokens/sec numbers and "this was a waste of time" stories are especially welcome.
Playing with the STM32MP2
Hacker NewsComments
I'm not convinced the AI bubble exists
Hacker NewsComments
Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving
arXiv cs.LGarXiv:2609.04748v3 Announce Type: replace-cross
Abstract: Major open-source serving stacks reuse the key-value tensors of a shared prefix by default. We measure what this prefix reuse costs in reproducibility. Holding model, decoding, seed and request order fixed at batch size one, we ran an eighty-episode multi-turn agentic tool-use workload with prefix reuse on and off across two engines and four weight formats. Without prefix reuse, repeated runs reproduced every token and log-probability in all ten configurations. With it, cache-hit and recompute paths diverged on both engines. On llama.cpp with GGUF weights this path divergence rises with quantization: for Qwen2.5-7B over ten independent server histories, it reached 37.4 percent of episodes at 16-bit and 82.5 percent at four-bit. Near-tie decisions became only about 1.2 times as common, while the cache-path change in top-two margins grew 4.9-fold from 16-bit to three-bit. Serving four-bit weights through 16-bit kernels left divergence not detectably different from 16-bit, placing the amplification mainly in the quantized execution path. At Q4_K_M, run-to-run divergence came mainly from llama.cpp's server-level prompt cache: under the canonical episode order it moved divergence by 37.5 points, and at its default, divergence spanned 20.0 to 28.7 percent under five random orders. With cache state restored, each path reproduced exactly while the two still differed. In the tested stacks, cached serving is irreproducible because the cache state is absent from the request and not reset by default. A single-turn bridge shows path divergence reaching task outcomes with no detectable net change in accuracy. We release the harness, logs, and analysis pipeline.
Measuring and Decomposing Mode Separation via the Canonical Diffusion
arXiv cs.LGarXiv:2605.08777v2 Announce Type: replace-cross
Abstract: Mode separation, namely how sharply a distribution fragments into barrier-separated clusters, is a fundamental geometric property of densities, difficult to quantify in high dimensions. Modes are often the meaningful alternatives in the data, such as the conformations of a molecule or the distinct semantic types among a generative model's outputs. Mode separation is distinct from dispersion, which is what standard label-free tools capture: differential entropy rises with spread regardless of fragmentation, and PCA orders directions by variance regardless of barriers. Mutual information does capture it, but only given the latent component labels, which are usually unavailable. We measure mode separation through a single stochastic process intrinsic to the density, namely the unique reversible diffusion with $f$ as its stationary distribution and a constant scalar diffusion coefficient. From its autocovariance matrix we extract two readouts. The first, SSA (Sum of Squared Autocorrelations), is a scalar barrier-sensitive measure. The second, DA (Dominant Autocorrelation directions), gives linear projections ordered by metastability rather than variance. Under an isotropic-Gaussian null, we derive the limiting spectrum of the empirical autocovariance in closed form, generalizing the Marchenko--Pastur law. Its upper edge gives a noise floor that separates metastable directions from sampling noise. Both readouts use only samples and a score function, scaling to high dimensions through pretrained score-based generative models via Tweedie's identity. We apply our framework to three settings: (i) synthetic Gaussian mixtures, where SSA tracks mutual information; (ii) SDXL text-to-image generations, where SSA and DA capture structure that entropy and PCA miss; and (iii) molecular dynamics of alanine dipeptide, where DA recovers the known slow backbone dihedrals from static samples alone.
The Sparsity Whisperer
arXiv cs.LGarXiv:2608.06630v2 Announce Type: replace
Abstract: Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs. We argue that this overlooks a key computation performed by particularly sparsity-sensitive neurons in the MLP up and gate projections: separating similar inputs into dissimilar outputs. This suggests that effective pruning should preserve not only activations, but also the differences between outputs more broadly. We introduce a family of difference-informed pruning methods built upon this principle. Wisp+ is a neuronwise, update-free method that prioritizes the input pairs each neuron separates most strongly relative to their distance. Wisp provides a more efficient layerwise alternative that scores weights using shared input-difference norms. Finally, Whisper extends global difference preservation to second-order reconstruction using a lightly regularized difference Hessian. Across Llama 2 and 3.1 models from 7B to 405B parameters, our second-order variant consistently improves over strong reconstruction-based baselines, while our update-free variants improve over activation-aware baselines, especially in constrained settings. The improvements over Wanda and SparseGPT extend to structured sparsity, downstream evaluations, and other model families. Augmenting stronger techniques such as RIA and ALPS with our difference-informed criteria yields further improvements, shifting the overall accuracy-runtime frontier outward at negligible additional cost. These results suggest that preserving output differences is a broadly useful and composable signal for post-training LLM sparsification.
Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs
arXiv cs.LGarXiv:2606.05516v2 Announce Type: replace
Abstract: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers. In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer. Across multiple LLM families and downstream tasks, fine-tuning this dominant layer alone consistently matches or even exceeds full-model ZO fine-tuning. We further show that the dominant layer is task-agnostic but model-specific, and can be identified before training through a simple inference-only analysis of activation outliers. Specifically, the dominant layer consistently aligns with the first activation-outlier layer in the pre-trained model. To explain this phenomenon, we analyze how perturbation effects propagate under ZO optimization. We find that the dominant layer combines two key properties: high perturbation sensitivity and early placement in the residual stream, allowing perturbation-induced effects to propagate and accumulate through remaining subsequent decoding layers. As a result, this layer produces disproportionately strong and stable optimization signals under forward-only updates. Extensive experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning while achieving up to 4.52$\times$ training speedup.
The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
arXiv cs.LGarXiv:2605.19537v3 Announce Type: replace
Abstract: Progress in LLMs is increasingly measured through standardized benchmarks, where state-of-the-art improvements are often separated by fractions of a percentage point. At the same time, the computational cost of evaluating modern LLMs has driven widespread adoption of specialized inference backends, software systems that execute trained models efficiently at inference time. While critical for scalability, system-level optimizations, such as custom CUDA kernels and reduced-precision arithmetic, can alter token probabilities and introduce non-determinism, possibly cascading into divergent generation. In this work, we first survey the inference landscape, identifying $200$ distinct engines, and analyze $35,000$ ML publications, finding that the specific inference stack is rarely reported despite this widespread diversity. We then present a systematic empirical study of how inference backends affect LLM benchmark results. Holding model weights, decoding parameters, and hardware constant, we evaluate five widely used inference engines, including vLLM, SGLang, and llama.cpp, across multiple open-weight models spanning 4B-30B parameters and established benchmarks. We show that the choice of backend alone can shift benchmark scores by up to $16.6$ percentage points and induce high rates of output disagreement. By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations such as prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing. Our findings identify the inference backend as a previously unreported yet consequential hyperparameter in LLM evaluations and advocate standardized reporting of inference stacks to improve the reproducibility and interpretability of benchmark comparisons.
What Does a Sparse Autoencoder Feature Do? A Weight-Based Account
arXiv cs.LGarXiv:2601.22447v2 Announce Type: replace
Abstract: Sparse autoencoders (SAEs) have emerged as a powerful technique for decomposing language model representations into interpretable features. Current interpretation pipelines infer feature semantics from activation patterns, implicitly assuming every feature has a semantic explanation while overlooking the computational roles features inherit from the SAE training objective. We introduce a weight-based interpretation framework that requires no activation data and grounds claims causally through targeted feature-ablation interventions. Across Gemma-2 and Llama-3.1, three depth-dependent signatures show that SAE features inherit the base model's geometry: semantic features form a U-shape under tied embeddings but bifurcate when untied, attention participation peaks mid-layer, and the two roles couple oppositely for input- vs. output-oriented features. This weight-based view supplies the causal half missing from activation-based interpretability.
Towards the Worst-case Robustness of Large Language Models
arXiv cs.LGarXiv:2501.19040v5 Announce Type: replace
Abstract: Recent studies have revealed the vulnerability of large language models to adversarial attacks, where adversaries craft specific inputs to induce incorrect or harmful outputs. Although various empirical defenses have been proposed, their worst-case robustness remains poorly understood, raising concerns about how they would behave against stronger future attacks. Here, we study the worst-case robustness of LLMs from both empirical and theoretical perspectives. First, we provide tighter empirical upper bounds for the tested deterministic defenses using stronger white-box attacks, under which most tested configurations achieve nearly 0\% robustness. Second, we formulate randomized-smoothing certification for bounded and binary base functions as fractional and 0-1 knapsack problems, respectively, and derive grouped solvers for two token kernels. For example, on AdvBench, we certify a GPT-4o safety detector smoothed by a uniform kernel to an average certified $\ell_0$ radius of 2.02. In our stated suffix-attack setting, the same kernel certifies a Llama-3-8B safety detector to an average certified radius of 6.41.
Personalized health coaching via coachable health age and counterfactual wearable trajectories
arXiv cs.LGarXiv:2611.08568v1 Announce Type: cross
Abstract: Personalized health coaching requires moving beyond retrospective wearable summaries toward quantitative, actionable behavioral prescriptions. We introduce Coachable Health Age (CHA), an interpretable health-age metric built from modifiable physiological and behavioral signals passively captured by consumer wearables. Across the WEAR-ME (N=2,780), All of Us (N=22,748), and longitudinal Fitbit (N=1,464) cohorts, CHA aligns with established cardiometabolic health constructs and improves prognostic discrimination for incident cardiovascular and metabolic disease beyond age, sex, and body mass index. To enable prospective behavioral optimization, we developed Sensor World Model (SensorWM), an action-conditioned wearable world model pretrained on over 2.2 million person-weeks of longitudinal sensor data from approximately 100,000 participants. SensorWM accurately forecasts multi-week physiological responses to candidate lifestyle changes ($R^2=0.664$), outperforming generic time-series foundation models while preserving downstream clinical signal for diabetes, hypertension, and other health outcomes. Finally, we integrate CHA and SensorWM into an agentic counterfactual coaching system to synthesize personalized 12-week coaching plans. In blinded paired reviews, certified lifestyle coaches preferred counterfactual-grounded plans over plans grounded in wearable history alone for overall quality (71.4%, P < 0.001) and anticipated adherence (71.0%, P < 0.001), and board-certified clinicians judged 99.2% of counterfactual-grounded plans to meet clinical safety criteria (vs. 97.6% of comparison plans). Automated evaluations across the full cohort (N=2,780) also favored counterfactual-grounded plans. By closing the loop between longitudinal tracking, trajectory prediction, and counterfactual coaching, our framework provides a scalable path to systematically improve cardiometabolic health.
Know What You Need: Efficient Activation Checkpointing
arXiv cs.LGarXiv:2611.09614v1 Announce Type: new
Abstract: Large Language Model training is increasingly constrained by memory capacity. Activation Checkpointing reduces activation memory by avoiding storing selected activations during the forward pass and rematerializing them during backpropagation, thereby trading memory savings for additional computation. Although this technique is essential for scaling model training, rematerialization can be prohibitively expensive, particularly for Transformer linear layers. Approximate computation could reduce this overhead, but conventional approaches either introduce errors that accumulate over the backward pass or fail to exploit information available from the initial forward pass. We introduce a novel approximate rematerialization method that can yield unbiased gradient estimates: using information collected during the initial forward pass, the method selectively recomputes only the most important layer components, either through greedy selection or randomized sketching. Ablation studies show that the resulting gradients closely match the exact gradients. When applied to feed-forward layers, attention layers, or both, our method can recompute only two-thirds of the layer components in a Llama-488M model, while increasing perplexity by just 1.7%, 1.2%, and 2.5%, respectively.
Transformer-FSQ: Discrete Latents for Extreme KV-Cache Compression
arXiv cs.LGarXiv:2611.09598v1 Announce Type: new
Abstract: Private, personalised language models must run on local consumer devices, where large, frontier models cannot fit, and retaining long contexts is non-trivial. Sequential memory---the key-value (KV) cache---grows linearly with context length, quickly dwarfing the model itself: for Qwen3-1.7B-Base, 200,000 retained entries require 21 GiB, 6.6 times more than the model weights. Yet the KV-cache compression methods we measured fail beyond moderate compression ratios. We introduce Transformer-FSQ, addressing extreme KV-cache compression by replacing standard key and value projections with a finite scalar quantisation bottleneck. Our cache stores a packed vector of discrete integer codes per token, with no per-token quantisation metadata. Converting a pretrained transformer trains only these projections on 0.25 tokens per original parameter. We evaluate Transformer-FSQ against five baseline families on models from 0.6B to 8B parameters. On long-context retrieval (RULER), from 12$\times$ to 97.5$\times$ Transformer-FSQ is the most accurate method we evaluate, across scales. A fused Triton kernel decodes directly from the integer codes; with baselines given their authors' kernels where available, it decodes at $1.01\times$ (95% interval $0.98$--$1.05$) the throughput of native SDPA attention on the uncompressed Qwen3-8B-Base at $32\times$ and $1.54\times$ at $97.5\times$. Extreme KV-cache compression is therefore a practical way to extend how much context a fixed device can hold.
MINE: Entropy-coded Memory-efficient Inference for Large Language Models
arXiv cs.LGarXiv:2611.09424v1 Announce Type: new
Abstract: We present MINE (Memory-efficient INference Engine), a lossless entropy coding approach that accelerates single-stream inference of Large Language Models (LLMs), where existing lossless compression methods yield speedups only for specific data types such as \texttt{BF16} and under additional assumptions. Single-stream decoding is strongly memory-bound, since every weight must be streamed from memory once per generated token and requests cannot be batched in local or low-latency deployments. At the core of MINE is dtANS-LLM, a GPU entropy coding scheme adapted from the prior dtANS to the needs of LLM inference. It exploits the non-uniform distribution of weights, which persists even after quantization, and exposes hyper-parameters that trade compression ratio against decoding cost. We auto-tune these parameters for each tensor and GPU to obtain the Pareto front between model size and decoding latency. MINE integrates into vLLM as a plugin and selects the best kernel configuration for a given memory budget. On Llama~3.1~8B~Instruct with a 2.1-bit EntQuant checkpoint, MINE achieves speedups of 2.52$\times$ and 1.48$\times$ over the fastest baseline on an RTX~4060~Ti and an RTX~5090, at effective bitrates of 2.24 and 3.43~bits per weight. MINE is available as open source.
Calibrated Uncertainty Quantification on TDC ADMET: What Can Be Resolved Across 22 Datasets?
arXiv cs.LGarXiv:2611.09373v1 Announce Type: new
Abstract: Absorption, distribution, metabolism, excretion, and toxicity (ADMET) models often rely on small, heterogeneous datasets, yet their predictions may determine which compounds proceed to experimental testing. Useful evaluations must therefore assess both predictive performance and the quality of reported probabilities or intervals. We ask which common uncertainty-quantification methods perform best on the 22 Therapeutics Data Commons (TDC) ADMET datasets and what this benchmark can resolve. We compare eleven calibrated pipelines using nested five-fold cross-validation across ten chemical partitions and five model seeds. Ensemble Monte Carlo (MC) dropout has the best average rank for TDC performance, Brier score, and 90% interval score. For TDC performance, however, it is not separated from the other ensemble pipelines; for Brier and interval score, LightGBM is also not ruled out as best. LightGBM wins more individual datasets than any other method for TDC performance and Brier score, but performs poorly on several endpoints. Smaller evaluations can identify a different leader and often fail to recover the broader comparison. These results make LightGBM and an ensemble neural model complementary baselines. Comparative claims should be checked across multiple chemical partitions using statistical analyses matched to the question being asked.
继续滚动加载更多…