Claudio Reggiani

Why Most Cross-Validation Visualizations Are Wrong (And How to Fix Them) https://ift.tt/3Im8iwz MODEL VALIDATION & OPTIMIZATION Stop using moving boxes to explain cross-validation! You know those cross-validation diagrams in every data science tutorial? The ones showing boxes in different colors moving around to explain how we split data for training and testing? Like this one: Have you seen that? Image by author. I’ve seen them too — one too many times. These diagrams are common — they’ve become the go-to way to explain cross-validation. But here’s something interesting I noticed while looking at them as both a designer and data scientist. When we look at a yellow box moving to different spots, our brain automatically sees it as one box moving around. It’s just how our brains work — when we see something similar move to a new spot, we think it’s the same thing. (This is actually why cartoons and animations work!) You might think the animated version is better, but now you can’t help following the blue box and starting to forget that this should represent how cross-validation works. Source: Wikipedia But here’s the thing: In these diagrams, each box in a new position is supposed to show a different chunk of data. So while our brain naturally wants to track the boxes, we have to tell our brain, “No, no, that’s not one box moving — they’re different boxes!” It’s like we’re fighting against how our brain naturally works, just to understand what the diagram means. Looking at this as someone who works with both design and data, I started thinking: maybe there’s a better way? What if we could show cross-validation in a way that actually works with how our brain processes information? All visuals: Author-created using Canva Pro. Optimized for mobile; may appear oversized on desktop. What’s Cross-Validation Really About? Cross-validation is about making sure machine learning models work well in the real world. Instead of testing a model once, we test it multiple times using different parts of our data. This helps us understand how the model will perform with new, unseen data. Here’s what happens: We take our data Divide it into groups Use some groups for training, others for testing Repeat this process with different groupings The goal is to get a reliable understanding of our model’s performance. That’s the core idea — simple and practical. (Note: We’ll discuss different validation techniques and their applications in another article. For now, let’s focus on understanding the basic concept and why current visualization methods need improvement.) What’s Wrong with Current Cross-validation Diagrams? Open up any machine learning tutorial, and you’ll probably see these types of diagrams: Long boxes split into different sections Arrows showing parts moving around Different colors showing training and testing data Multiple versions of the same diagram side by side Currently, this is similar to the first image you’ll see if...

👉🏼 ModelHamiltonian: A Python-scriptable library for generating 0-, 1-, and 2-electron integrals 🤓 Valerii Chuiko 👇🏻 https://lnkd.in/eeHkaTrr 🔍 Focus on data insights: - 📊 The ModelHamiltonian library allows seamless integration of electronic structure calculations with spin and occupation-based Hamiltonians. - ⚙️ Users can construct Hamiltonians through both programming and an intuitive graphical user interface, greatly expanding accessibility. - 🛠️ The inclusion of an interface to ChatGPT simplifies the specification of Hamiltonians, making advanced computational methods more approachable. 💡 Main outcomes and implications: - 🔑 The official release of ModelHamiltonian represents a significant step forward in facilitating quantum chemistry research and education. - 🌍 Its open-source nature promotes collaboration and innovation within the scientific community, enabling researchers to test and develop new methods more effectively. - 🎓 The user-friendly design fosters wider adoption in educational settings, enhancing the learning experience for students new to quantum chemistry. 📚 Field significance: - 🔬 This tool bridges a gap between traditional quantum chemistry software and modern computational methods, potentially leading to breakthroughs in material science and condensed matter physics. - 🚀 The library's design principles emphasize reproducibility and reliability, which are critical for scientific validation. - 🌐 By fostering user engagement through interactive interfaces, ModelHamiltonian can stimulate interest and research activity in computational chemistry among diverse populations. 🗄️: [#ModelHamiltonian #QuantumChemistry #PythonLibrary #ElectronicStructure #OpenSource #UserInterface #ComputationalPhysics #EducationalTools]

A comparative study of conformal prediction methods for valid uncertainty quantification in machine learning [PhD thesis] Nicolas Dewolf Abstract In the past decades, most work in the area of data analysis and machine learning was focused on optimizing predictive models and getting better results than what was possible with existing models. To what extent the metrics with which such improvements were measured were accurately capturing the intended goal, whether the numerical differences in the resulting values were significant, or whether uncertainty played a role in this study and if it should have been taken into account, was of secondary importance. Whereas probability theory, be it frequentist or Bayesian, used to be the gold standard in science before the advent of the supercomputer, it was quickly replaced in favor of black box models and sheer computing power because of their ability to handle large data sets. This evolution sadly happened at the expense of interpretability and trustworthiness. However, while people are still trying to improve the predictive power of their models, the community is starting to realize that for many applications it is not so much the exact prediction that is of importance, but rather the variability or uncertainty. The work in this dissertation tries to further the quest for a world where everyone is aware of uncertainty, of how important it is and how to embrace it instead of fearing it. A specific, though general, framework that allows anyone to obtain accurate uncertainty estimates is singled out and analysed. Certain aspects and applications of the framework -- dubbed `conformal prediction' -- are studied in detail. Whereas many approaches to uncertainty quantification make strong assumptions about the data, conformal prediction is, at the time of writing, the only framework that deserves the title `distribution-free'. No parametric assumptions have to be made and the nonparametric results also hold without having to resort to the law of large numbers in the asymptotic regime. 👉 https://lnkd.in/ddbwgukM #machinelearning

👉🏼 Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis 🤓 Mikaël Chelli 👇🏻 https://lnkd.in/e6jViaXK 🔍 Focus on data insights: - LLMs like ChatGPT and Bard were assessed for their performance in replicating human-conducted systematic reviews. - Precision rates varied significantly among the models tested. - Recall rates showed differences in the ability to retrieve relevant papers. - Hallucination rates were notably high, indicating the need for further refinement in LLM training. 💡 Main outcomes and implications: - LLMs should not be solely relied upon for conducting systematic reviews due to current performance limitations. - Validation of references generated by LLMs is crucial for research accuracy. - High hallucination rates emphasize the importance of enhancing LLM training before widespread academic use. 📚 Field significance: - Automated literature search tools like LLMs have potential but require refinement for reliable systematic review processes. - Addressing hallucination rates in LLMs is essential for maintaining research integrity. 🗄️: [#hallucinationrates #referenceaccuracy #systematicreviews #LLMs #researchvalidation]

📃Scientific paper: Croissant: A Metadata Format for ML-Ready Datasets Abstract: Data is a critical resource for Machine Learning (ML), yet working with data remains a key friction point. This paper introduces Croissant, a metadata format for datasets that simplifies how data is used by ML tools and frameworks. Croissant makes datasets more discoverable, portable and interoperable, thereby addressing significant challenges in ML data management and responsible AI. Croissant is already supported by several popular dataset repositories, spanning hundreds of thousands of datasets, ready to be loaded into the most popular ML frameworks. ;Comment: Published in Proceedings of ACM SIGMOD/PODS'24 Data Management for End-to-End Machine Learning (DEEM) Workshop https://lnkd.in/eGERJFs6 Continued on ES/IODE ➡️ https://etcse.fr/Msut ------- If you find this interesting, feel free to follow, comment and share. We need your help to enhance our visibility, so that our platform continues to serve you.

Latest NPS Hacker Cup lecture featuring Krista, a second-year PhD candidate at Stanford University working on DSPy! 🎉 In this session, Krista dives into how DSPy is transforming code generation by enabling developers to program rather than prompt language models. She explains how DSPy helps build and optimize language model pipelines composed of modular LM calls, making complex tasks more efficient and scalable. Key highlights: The advantage of modular LM pipelines over single LM calls in code generation. How DSPy separates program logic from LM instruction, reducing manual prompt engineering. An overview of DSPy's optimizers for refining prompts and weights automatically. A hands-on tutorial using DSPy with the Hacker Cup dataset. If you're looking to enhance your AI toolkit and streamline your code generation processes, this lecture is a must-watch! 🚀 #AI #DSPy #CodeGeneration #MachineLearning #LanguageModels #NPSHackerCup #Stanford

👉🏼 A comparison of human, GPT-3.5, and GPT-4 performance in a university-level coding course 🤓 Will Yeadon 👇🏻 https://lnkd.in/eP3vth6e 🔍 Focus on data insights: - 📈 Students achieved an average score of 91.9%, outperforming the best AI submissions, which scored 81.1%. - 🤖 Prompt engineering significantly enhanced the performance of both GPT-4 and GPT-3.5 models. - 🏷️ The accuracy of human evaluators in identifying authorship was 85.3%, indicating that AI-generated work can often be distinguished from human work. 💡 Main outcomes and implications: - 🎓 The study highlights that while AI can produce high-quality work, it is still inferior to the capabilities of university students in complex tasks. - 🚀 The effectiveness of prompt engineering suggests that tailored input can substantially improve AI performance, thus opening avenues for further research in enhancing AI systems. - 🔍 The ability of human markers to identify AI submissions raises ethical considerations regarding the use of AI in academic settings. 📚 Field significance: - 📊 This research contributes to understanding the capabilities and limitations of current AI technologies in educational environments. - ⚖️ It provides insights into the emerging dynamics of human-AI collaboration, particularly in academic assessments. - 🧠 The findings inform stakeholders about the challenges and potential strategies for integrating AI responsibly in educational frameworks. 🗄️: [#performance #AI #GPT3.5 #GPT4 #education #assessment #promptengineering #humanAIcollaboration]