Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member institutions, and all contributors. Donate
arxiv logo > stat > arXiv:0712.0248

Help | Advanced Search

Statistics > Machine Learning

(stat)
[Submitted on 3 Dec 2007]

Title:Pac-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning

Authors:Olivier Catoni
View a PDF of the paper titled Pac-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning, by Olivier Catoni
View PDF
Abstract: This monograph deals with adaptive supervised classification, using tools borrowed from statistical mechanics and information theory, stemming from the PACBayesian approach pioneered by David McAllester and applied to a conception of statistical learning theory forged by Vladimir Vapnik. Using convex analysis on the set of posterior probability measures, we show how to get local measures of the complexity of the classification model involving the relative entropy of posterior distributions with respect to Gibbs posterior measures. We then discuss relative bounds, comparing the generalization error of two classification rules, showing how the margin assumption of Mammen and Tsybakov can be replaced with some empirical measure of the covariance structure of the classification this http URL show how to associate to any posterior distribution an effective temperature relating it to the Gibbs prior distribution with the same level of expected error rate, and how to estimate this effective temperature from data, resulting in an estimator whose expected error rate converges according to the best possible power of the sample size adaptively under any margin and parametric complexity assumptions. We describe and study an alternative selection scheme based on relative bounds between estimators, and present a two step localization technique which can handle the selection of a parametric model from a family of those. We show how to extend systematically all the results obtained in the inductive setting to transductive learning, and use this to improve Vapnik's generalization bounds, extending them to the case when the sample is made of independent non-identically distributed pairs of patterns and labels. Finally we review briefly the construction of Support Vector Machines and show how to derive generalization bounds for them, measuring the complexity either through the number of support vectors or through the value of the transductive or inductive margin.
Comments: Published in at this http URL the IMS Lecture Notes Monograph Series (this http URL) by the Institute of Mathematical Statistics (this http URL)
Subjects: Machine Learning (stat.ML)
MSC classes: 62H30, 68T05, 62B10 (Primary)
Report number: IMS-LNMS56-LNMS5601
Cite as: arXiv:0712.0248 [stat.ML]
  (or arXiv:0712.0248v1 [stat.ML] for this version)
  https://doi.org/10.48550/arXiv.0712.0248
arXiv-issued DOI via DataCite
Journal reference: IMS Lecture Notes Monograph Series 2007, Vol. 56, i-xii, 1-163
Related DOI: https://doi.org/10.1214/074921707000000391
DOI(s) linking to related resources

Submission history

From: Olivier Catoni [view email] [via VTEX proxy]
[v1] Mon, 3 Dec 2007 13:49:36 UTC (182 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled Pac-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning, by Olivier Catoni
  • View PDF
  • TeX Source
  • Other Formats
view license
Current browse context:
stat.ML
< prev   |   next >
new | recent | 2007-12
Change to browse by:
stat

References & Citations

  • NASA ADS
  • Google Scholar
  • Semantic Scholar
a export BibTeX citation Loading...
PAC-BAYESIAN SUPERVISED CLASSIFICATION: ...
  • O. Catoni
Semantic ScholarArXiv articleJournal articleGoogle Scholar
Citation entry
Bibex: Semantic Scholar

Bookmark

BibSonomy logo Reddit logo

Bibliographic and Citation Tools

Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)

Bibliographic data

Select data provider: Semantic Scholar|Prophy[Disable Bibex (What is Bibex?)]
References (41)
Data provided by:
(report data issues)
Semantic Scholar
▲▼
Pages:
  • ◀
  • 1
  • 2
  • 3
  • 4
  • 5
  • ▶
Citation entry
Aggregated estimators and empirical complexity for least square regression2004
Jean-Yves Audibert
View article:
Semantic ScholarJournal articleGoogle Scholar
Citation entry
Optimal aggregation of classifiers in statistical learning2003
A. Tsybakov
View article:
Semantic ScholarJournal articleGoogle Scholar
Citation entry
Scale-sensitive dimensions, uniform convergence, and learnabilityProceedings of 1993 IEEE 34th Annual Foundations of Computer Science1993
N. AlonS. Ben-DavidN. Cesa-BianchiD. Haussler
View article:
Semantic ScholarJournal articleGoogle Scholar
Citation entry
Statistical learning theory1998
V. Vapnik
View article:
Semantic ScholarJournal articleGoogle Scholar
Citation entry
Adaptive context trees and text clusteringIEEE Trans. Inf. Theory2001
Jean-Philippe Vert
View article:
Semantic ScholarJournal articleGoogle Scholar
An Improved Predictive Accuracy Bound for Averaging ClassifiersICML2001
J. LangfordM. SeegerN. Megiddo
View article:
Semantic ScholarGoogle Scholar
Citation entry
An Introduction to Support Vector Machines and Other Kernel-Based Learning MethodsAI Mag.2001
Tong Zhang
View article:
Semantic ScholarJournal articleGoogle Scholar
Applications of empirical process theory2000
S. Geer
View article:
Semantic ScholarGoogle Scholar
Citation entry
Are Bayes Rules Consistent in Information1987
A. Barron
View article:
Semantic ScholarJournal articleGoogle Scholar
Bounds for averaging classifiers2001
View article:
Google Scholar
Citations (448)
Data provided by:
(report data issues)
Semantic Scholar
▲▼
Pages:
  • ◀
  • 1
  • 2
  • 3
  • 4
  • 5
  • ...
  • 45
  • ▶
A Column Generation Bound Minimization Approach with PAC-Bayesian Generalization GuaranteesAISTATS2016
Jean-Francis RoyM. MarchandFrançois Laviolette
View article:
Semantic ScholarGoogle Scholar
Citation entry
A Generalization Bound for Online Variational InferenceACML2019
Badr-Eddine Chérief-AbdellatifPierre AlquierM. E. Khan
View article:
Semantic ScholarArXiv articleGoogle Scholar
Citation entry
A New Family of Generalization Bounds Using Samplewise Evaluated CMINeurIPS2022
Fredrik HellströmG. Durisi
View article:
Semantic ScholarArXiv articleJournal articleGoogle Scholar
A New PAC-Bayesian View of Domain AdaptationNIPS 20152015
Pascal GermainFrançois LavioletteAmaury HabrardEmilie Morvant
View article:
Semantic ScholarGoogle Scholar
A PAC-Bayes Sample-compression Approach to Kernel MethodsICML2011
Pascal GermainAlexandre LacosteFrançois LavioletteM. MarchandSara Shanian
View article:
Semantic ScholarGoogle Scholar
Citation entry
A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient DescentNIPS2017
Ben London
View article:
Semantic ScholarArXiv articleGoogle Scholar
A PAC-Bayesian Approach to Structure Learning2009
Yevgeny Seldin
View article:
Semantic ScholarGoogle Scholar
Citation entry
A PAC-Bayesian Link Between Generalisation and Flat MinimaALT2024
Maxime HaddouchePaul ViallardUmut SimsekliBenjamin Guedj
View article:
Semantic ScholarArXiv articleJournal articleGoogle Scholar
Citation entry
A PAC-Bayesian Tutorial with A Dropout BoundArXiv2013
David A. McAllester
View article:
Semantic ScholarArXiv articleGoogle Scholar
Citation entry
A Primer on PAC-Bayesian LearningICML 20192019
Benjamin Guedj
View article:
Semantic ScholarArXiv articleGoogle Scholar

Connected Papers

Example graph image

See related papers to:

PAC-BAYESIAN SUPERVISED CLASSIFICATION: The Thermodynamics of Statistical Learning

Placeholder image of Literature Map for “PAC-BAYESIAN SUPERVISED CLASSIFICATION: The Thermodynamics of Statistical Learning”

Build Literature Map from “PAC-BAYESIAN SUPERVISED CLASSIFICATION: The Thermodynamics of Statistical Learning”

Litmaps Logo
publications
3
supporting
0
mentioning
2
contrasting
0
Smart Citations
3
0
2
0
Citing PublicationsSupportingMentioningContrasting
View Citations

See how this article has been cited at scite.ai

scite shows how a scientific paper has been cited by providing the context of the citation, a classification describing whether it supports, mentions, or contrasts the cited claim, and a label indicating in which section the citation was made.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
  • About
  • Help
  • contact arXivClick here to contact arXiv Contact
  • subscribe to arXiv mailingsClick here to subscribe Subscribe
  • Copyright
  • Privacy Policy
  • Web Accessibility Assistance
  • arXiv Operational Status
    Get status notifications via email or slack