Combine Classification Agorithms



is it possible to combine logistic regression, SVM, decision tree for classification


Hi Erigits,

on High level answer inline.

Multiple classifiers - Definition.
• Multiple classifier – a set of classifiers whose individual
predictions are combined in some way to classify new
• Various names: ensemble methods, committee, classifier
fusion, combination, aggregation,…
• Integration should improve predictive accuracy.

Why could we integrate classifiers?
• Typical research → create and evaluate a single learning
algorithm; compare performance of some algorithms.
• Empirical observations or applications → a given algorithm
may outperform all others for a specific subset of problems
•There is no one algorithm achieving the best accuracy for all
• A complex problem can be decomposed into multiple subproblems
that are easier to be solved.
• Growing research interest in combining a set of learning
algorithms / classifiers into one system
„Multiple learning systems try to exploit the local
different behavior of the base learners to enhance
the accuracy of the overall learning system”

Multiple classifiers – why do they work?

• How to create such systems and when they may perform
better than their components used independently?
• Combining identical classifiers is useless!
A necessary condition for the approach to be useful is
that member classifiers should have a substantial level of
disagreement, i.e., they make error independently with
respect to one another

Why do ensembles work?

• The Statistical Problem arises when the hypothesis space is too large
for the amount of available data. Hence, there are many hypotheses with
the same accuracy on the data and the learning algorithm chooses only
one of them! There is a risk that the accuracy of the chosen hypothesis is
low on unseen data!
• The Computational Problem arises when the learning algorithm cannot
guarantee finding the best hypothesis.
• The Representational Problem arises when the hypothesis space does
not contain any good approximation of the target classes

Hope this helps



Hi @erigits, in addition to @tillutony I would recommend you read this blog on ensemble learning.