Network quantization can effectively reduce computation and memory costs, facilitating the deployment of complex Deep Neural Networks (DNNs) on mobile equipment. However, the low-bit quantization without time-consuming training or access to the full datasets is still a challenging problem. In this paper, we develop a two-stage quantization method to address these issues, which only requires a few unlabeled samples. Firstly, we present a gradient-based approach to analyze per-channel sensitivity and optimize the bit-width allocation for different channels according to their sensitivity. Secondly, we propose to refine the quantization model by distilling knowledge from the output and intermediate features of the pre-trained model. Extensive experiments on image classification and object detection demonstrate the effectiveness of the proposed method, and it can achieve a promising result in 4-bit quantization.