Sec-News 信息安全播报平台数据统计

地址：http://wiki.ioin.in/
Date：2016-12-17
0x00 前言
一直在关注着这个平台推送的信息，很喜欢，但是一直想知道，他们都是从哪里获得的信息，于是就有了今天的这篇文章。

整体的思路有:

• 批量获取短域名
• 解析短域名
• 分解URL，统计Host
• Sort排序
  

个别网址需要翻墙，本次未做统计
0x01 Just Do Your Work
1.批量获取短域名,

[Python] 纯文本查看 复制代码

import requests
import re

if __name__ == '__main__':
	headers = {
	'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
	'Accept-Encoding':'gzip, deflate, sdch',
	'Accept-Language':'zh-CN,zh;q=0.8',
	'Cache-Control':'max-age=0',
	'Host':'wiki.ioin.in',
	'Connection':'keep-alive',
	'Upgrade-Insecure-Requests':'1',
	'User-Agent':'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36'}
	for i in xrange(1,85):#85
		url = 'http://wiki.ioin.in/page-{}'.format(i)
		try:
			req = requests.get(url,headers=headers)
			flag = re.findall(r'<a href="(.*?)" class="visit-color" target=',req.content)
			for j in xrange(len(flag)):
				print len(flag)
				with open('tar.txt','a')as tar:
					tar.write('http://wiki.ioin.in')
					tar.write(flag[j])
					tar.write('\n')
		except:
			pass

2.短域名解析

[Python] 纯文本查看 复制代码

import requests

def main(szURL,timeout=5):
    headers = {
    'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Encoding':'gzip, deflate',
    'Accept-Language':'zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3',
    'Upgrade-Insecure-Requests':'1',
    'Connection': 'close',
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0'}
    s = requests.Session()
    try:
        req = s.get(szURL,headers=headers,timeout=timeout)
        print req.url
        with open('result.txt','a')as tar:
            tar.write(req.url)
            tar.write('\n')
    except:
        pass

if __name__ == "__main__":
     f = open('tar.txt','r')
     for i in f.readlines():
        szURL =  i.strip('\n')
        main(szURL)

3.分解URL，统计Host

[Python] 纯文本查看 复制代码

def deal(i):
	d = i.split('/')[2]
	with open('new_result.txt','a')as tar:
		tar.write(d)
		tar.write('\n')

def main():
	f = open('result.txt','r')
	for i in f.readlines():
		deal(i)

if __name__ == '__main__':
	main()

4.Sort 排序

[Python] 纯文本查看 复制代码

import re
import collections

def count_word(path):
    result = {}
    with open(path) as file_obj:
        all_the_text = file_obj.read()
        all_the_text = all_the_text.lower()
        #all_the_text = re.sub("\"|,|\.", "", all_the_text)
        for word in all_the_text.split():
            if word not in result:
                result[word] = 0
            result[word] += 1       
    return result
    
def sort_by_count(d):
    d = collections.OrderedDict(sorted(d.items(), key = lambda t: -t[1]))
    return d

if __name__ == '__main__':
    file_name = "new_result.txt"
    dword = count_word(file_name)
    dword = sort_by_count(dword) 
    for key,value in dword.items():
        print key + ":%d" % value        
        with open('sort_result.txt','a')as tar:
            tar.write(key)
            tar.write(',')
            tar.write(str(value))
            tar.write('\n')

5.Top 10

[Shell] 纯文本查看 复制代码

docs.ioin.in	    63
github.com	        55
bobao.360.cn	    52
[url]www.freebuf.com[/url]  	45
mp.weixin.qq.com	40
segmentfault.com	38
paper.seebug.org	29
[url]www.leavesongs.com[/url]	23
wiki.ioin.in	    20
[url]www.cnblogs.com[/url]	    19

6.All Data
纸上得来终觉浅，绝知此事要躬行