In order to meet stringent environmental, safety and performance requirements from respective regulatory bodies, various technology-based industries are promoting the use of advanced carbon nanotube (CNT) reinforced lightweight and high strength polymer nanocomposites (PNCs) as a substitute to conventional materials both in structural and non-structural applications. The superior mechanical properties of PNCs made up of CNTs or bundles of CNTs can be attributed to the interfacial interaction between the CNTs and matrix, CNT's morphologies and to their uniform dispersion in the matrix. In PNCs, CNTs physically bond with polymeric matrix at a level where the assumption of continuum level interactions is not applicable. Modeling and prediction of mechanical response and failure behavior of CNTs and their composites becomes a complex task and is dealt with the help of up-scale modeling strategies involving multiple spatial and temporal scales in hierarchical or concurrent manner. Firstly, the article offers an insight into various modeling techniques in studying the mechanical response of CNTs; namely, equivalent continuum approach, quasi-continuum approach and molecular dynamics (MD) simulation. In the subsequent steps, these approaches are combined with analytical and numerical micromechanics models in a multiscale framework to predict the average macroscopic response of PNCs. The review also discusses the implementation aspects of these computational approaches, their current status and associated challenges with a future outlook.