2020-04
26

算算书里有多少单词

By xrspook @ 18:12:57 归类于: 扮IT

算算书里有多少单词应该是很大路简单的事，但实际上各种状况层出不穷。有些是你料到的，比如排版的用了全角的标点符号，程序默认会删掉标点符号，万一排版那个没有规范地使用空格呢？有些是你不会料到的，比如手误创造出奇葩字符串。很早以前我就发现Notepad++和Word里算的字数是不一致的，Notepad++通常算出来的数都会大一些。谁对谁错，随缘吧，知道大概差不多也就行了，毕竟高考的时候你写少几个字不到800也不会真扣你的分。

字典和列表的相爱相杀我体会得越来越深刻了。

words.txt在这里，emma.txt在这里。

Exercise 1: Write a program that reads a file, breaks each line into words, strips whitespace and punctuation from the words, and converts them to lowercase. Hint: The string module provides a string named whitespace, which contains space, tab, newline, etc., and punctuation which contains the punctuation characters. Let’s see if we can make Python swear:
>>> import string
>>> string.punctuation
‘!”#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~’
Also, you might consider using the string methods strip, replace and translate.

Exercise 2: Go to Project Gutenberg (http://gutenberg.org) and download your favorite out-of-copyright book in plain text format. Modify your program from the previous exercise to read the book you downloaded, skip over the header information at the beginning of the file, and process the rest of the words as before. Then modify the program to count the total number of words in the book, and the number of times each word is used. Print the number of different words used in the book. Compare different books by different authors, written in different eras. Which author uses the most extensive vocabulary?

Exercise 3: Modify the program from the previous exercise to print the 20 most frequently used words in the book.

Exercise 4: Modify the previous program to read a word list (see Section 9.1) and then print all the words in the book that are not in the word list. How many of them are typos? How many of them are common words that should be in the word list, and how many of them are really obscure?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128

import string
fin = open('words.txt')
mydict = {}
for line in fin:
    word = line.strip()
    mydict[word] = ''
file = open('emma.txt', encoding = 'utf-8')
essay = file.read().lower()
essay = essay.replace('-', ' ')
pun = {}
str_all = '“' + '”' + string.punctuation
for x in str_all: # 建立各种标点符号字符的字典
    pun[x] = ''
useless = essay.maketrans(pun) # maketrans必须被替换和替换等长，字典完美解决这个问题
l = essay.translate(useless).split() # 那些含-的单词会死得很惨，但仍然算是个单词
print('this book has', len(l), 'words')
book = {}
for item in l: # 读取文件为字符串，字符串转为单词列表，列表转为计数的字典，单词为键，次数为键值
    book[item] = book.get(item, 0) + 1
list_words1 = sorted(list(zip(book.values(), book.keys())), reverse = True) # 字典转为列表，键与键值换位
print('this book has', len(list_words1), 'different words')
print('times', 'word', sep='\t')
count = 1
word_len = 0 # 限制最小词长
for times, word in list_words1: # 打印大于某长度用得最多的20个词（不限制，3个字母及以下最最简单的会刷屏）
    if len(word) > word_len:
        print(times, word, sep='\t')
        count += 1
    if count > 20:
        break
count = 0
for word in book:
    if word not in mydict:
        # print(word, end=' ')
        count += 1
print(count, 'words in book not in dict') # 结果惨不忍睹，合计590个
# this book has 164065 words
# this book has 7479 different words
# times   word
# 5379    the
# 5322    to
# 4965    and
# 4412    of
# 3191    i
# 3187    a
# 2544    it
# 2483    her
# 2401    was
# 2365    she
# 2246    in
# 2172    not
# 2069    you
# 1995    be
# 1815    that
# 1813    he
# 1626    had
# 1448    as
# 1446    but
# 1373    for
# 590 words in book not in dict
# -----------------------------解法二----------------------------- 其实就是切单词方法有差异
import string
def set_book(fin1):
    useless = string.punctuation + string.whitespace + '“' + '”'
    d = {}
    for line in fin1:
        line = line.replace('-', ' ')
        for word in line.split():
            word = word.strip(useless)
            word = word.lower()
            d[word] = d.get(word, 0) + 1
    return d
def set_dict(fin2):
    d = {}
    for line in fin2:
        word = line.strip()
        d[word] = d.get(word, 0) + 1
    return d
fin1 = open('emma.txt', encoding='utf-8')
fin2 = open('words.txt')
book = set_book(fin1)
mydict = set_dict(fin2)
l = sorted(list(zip(book.values(), book.keys())), reverse=True)
count = 0
for key in book:
    count = count + book[key]
print('this book has', count, 'words')
print('this book has', len(book), 'different words')
num = 20
print(num, 'most common words in this book')
print('times', 'word', sep='\t')
for times, word in l:
    print(times, word, sep='\t')
    num -= 1
    if num < 1:
        break
count = 0
for word in book:
    if word not in mydict:
        # print(word, end=' ')
        count += 1
# print()
print(count, 'words in book not in dict')
# this book has 164120 words
# this book has 7531 different words
# 20 most common words in this book
# times   word
# 5379    the
# 5322    to
# 4965    and
# 4412    of
# 3191    i
# 3187    a
# 2544    it
# 2483    her
# 2401    was
# 2364    she
# 2246    in
# 2172    not
# 2069    you
# 1995    be
# 1815    that
# 1813    he
# 1626    had
# 1448    as
# 1446    but
# 1373    for
# 683 words in book not in dict
标签：, , , , , , , ,
还没有评论
2020-04
24

用两天琢磨一道题

By xrspook @ 20:32:30 归类于: 扮IT

前面还在沾沾自喜我写出来的脚本运行效率战胜了参考答案，但这道题目我是看着参考答案都不知道他们在说什么。如果只是一个词，我的确可以列举出它一次减少一个字母可以出现的所有可能，但怎么知道上一层可能和这一层的哪个配套？？？我花了2天时间去研究、消化答案。一边搞清楚答案为什么这样，另一边考虑有没有其它容易吃透的表达方式。这道题之所以让我非常纠结，根本的原因是我想不透到底我可以用什么手段实现。没有可以实现的逻辑，就不会有可行的编程。

Exercise 4: Here’s another Car Talk Puzzler (http://www.cartalk.com/content/puzzlers): What is the longest English word, that remains a valid English word, as you remove its letters one at a time? Now, letters can be removed from either end, or the middle, but you can’t rearrange any of the letters. Every time you drop a letter, you wind up with another English word. If you do that, you’re eventually going to wind up with one letter and that too is going to be an English word—one that’s found in the dictionary. I want to know what’s the longest word and how many letters does it have? I’m going to give you a little modest example: Sprite. Ok? You start off with sprite, you take a letter off, one from the interior of the word, take the r away, and we’re left with the word spite, then we take the e off the end, we’re left with spit, we take the s off, we’re left with pit, it, and I. Write a program to find all words that can be reduced in this way, and then find the longest one. This exercise is a little more challenging than most, so here are some suggestions: You might want to write a function that takes a word and computes a list of all the words that can be formed by removing one letter. These are the “children” of the word. Recursively, a word is reducible if any of its children are reducible. As a base case, you can consider the empty string reducible. The wordlist I provided, words.txt, doesn’t contain single letter words. So you might want to add “I”, “a”, and the empty string. To improve the performance of your program, you might want to memoize the words that are known to be reducible. Solution: http://thinkpython2.com/code/reducible.py.

最终，我觉得自己总算消化了，顺便画了个思维导图帮助大家理解到底分解到什么程度叫做完成，什么状态叫做分解失败。[”]和[]是两种不同的东西！！！！！！

is_reducible()是最关键的函数，memos用在这里，memos初始设置了known[”] = [”]也很关键，这是个守卫模式，没有守卫is_reducible()根本没法玩。这个脚本里的5个函数，除了一开始的创建字典函数，其余函数都可以单独测试，把一个固定单词放进去脚手架测试，可以帮助理解。cut_letter()，is_reducible()和all_reducible()这三个函数最终返回的都是列表，它们的样式都是类似的。希望我理解过程中的注释能帮助到有需要的人。PS一句：参考答案的打印效果让人很晕，我修改版的打印效果很美丽:)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103

from time import time
def set_dict(fin):
    d = {}
    for line in fin:
        word = line.strip()
        d[word] = 0
    for word in ['a', 'i', '']:
        d[word] = 0
    return d
def cut_letter(word, d): # 生成子单词，返回列表
    l = []
    for i in range(len(word)):
        new_word = word[:i] + word[i+1:]
        if new_word in d:
            l.append(new_word)
    return l # ['']长度为1，[]长度为0，无子词不能分解时返回[]，'a'返回['']
def is_reducible(word, d): # 判断能否生成无限子单词，返回列表
    if word in known: # 守卫模式下，''空字符串被列入初始字典，不列入永远会被递归到[]，无果
        return known[word]
    # if word == '': # 不用memos的时候，需要加入这句守卫
    #     return ['']
    l = []
    for new_word in cut_letter(word, d):
        if len(is_reducible(new_word, d)) > 0:
            l.append(new_word)
    known[word] = l
    return l
def all_reducible(d): # 收集所有无限子单词的单词，返回列表
    l = []
    for word in d:
        if len(is_reducible(word, d)) > 0: # 有列表，即有无限子单词
            l.append((len(word), word)) # 列表含有N个元组，元组里有2个元素，1为单词的字母数量，2为单词本尊
    new_l = sorted(l, reverse = True) # 每次减少一个字母，单词的字母越多当然就能降解出越多层了
    return new_l
def word_list(word): # 打印单词及子单词
    if len(word) == 0: # 最后一个进入is_reducible()的是['']，对应l[0]为无，打印结束
        return
    print(word)
    l = is_reducible(word, d) # 因为是被鉴定过词汇表里的词，所以必定有无限子单词
    word_list(l[0]) # 子单词有多个时只选第1个
known = {} # memos实际上只在is_reducible()起作用，除了提高效率，还能用作守卫
known[''] = [''] # 因为is_reducible()返回的是列表，所以即便是空字符串，键值也必须是列表！
fin = open('words.txt')
start = time()
d = set_dict(fin) # 普通的字典，键为单词，键值为0
words = all_reducible(d) # 列表，元组，2元素
for i in range(5):
    word_list(words[i][1]) # 列表里第某个元组的第2个元素
end = time()
print(end - start)
# complecting
# completing
# competing
# compting
# comping
# coping
# oping
# ping
# pig
# pi
# i
# twitchiest
# witchiest
# withiest
# withies
# withes
# wites
# wits
# its
# is
# i
# stranglers
# strangers
# stranger
# strange
# strang
# stang
# tang
# tag
# ta
# a
# staunchest
# stanchest
# stanches
# stances
# stanes
# sanes
# anes
# ane
# ae
# a
# restarting
# restating
# estating
# stating
# sating
# sting
# ting
# tin
# in
# i
# 0.6459996700286865
# 无memos 1.5830001831054688， 有memos 0.6459996700286865
标签：, , , , , , , , , ,
还没有评论
2020-04
22

字典还能这样玩！

By xrspook @ 18:30:55 归类于: 扮IT

一开始，我自己写的脚本能运行，但慢到怀疑人生。吃了个饭，折腾了半个小时后，字母表才处理到b而已，显然这是个失败的操作。我的做法是常规地为词汇表建立字典，然后历遍字典里的每个单词，单词进入函数后跟字典的另一个单词比较，比较方法是把单词（即字符串）打散为字符列表然后排列，如果排列一致，且被比较的单词小于拿去比的单词，它们就是一伙的，贴在被比较的单词列表下。列表长度大于2就返回列表然后打印。这样是可以选出异构词的，但非常非常慢！

看过参考答案之后我跳起来了，他们用了一句”.join(lists)，这等于是把列表str重新粘成一个字符串，我那个去！他们把单词用列表打散重排再粘回去，最关键的是，这个唯一的重排字符串他们在建立字典的时候就作为key，所有与之有一样字符的全部被看作小弟被放置这个键的键值里。字典还是字典，但字典的键成了规则字符串，键值则是排列组合过的词汇表。我根本没想到啊，怎么可能想得到呢！！！！！

题目要求倒序打印，然后要求找出能组成最多异构词的8个字母。但实际上参考答案的输出问非所答，比如没有倒序，比如只是把8个字母的异构词摆出来，没确切告诉你最多的是什么。

Exercise 2: More anagrams! Write a program that reads a word list from a file (see Section 9.1) and prints all the sets of words that are anagrams. Here is an example of what the output might look like:
[‘deltas’, ‘desalt’, ‘lasted’, ‘salted’, ‘slated’, ‘staled’]
[‘retainers’, ‘ternaries’]
[‘generating’, ‘greatening’]
[‘resmelts’, ‘smelters’, ‘termless’]
Hint: you might want to build a dictionary that maps from a collection of letters to a list of words that can be spelled with those letters. The question is, how can you represent the collection of letters in a way that can be used as a key? Modify the previous program so that it prints the longest list of anagrams first, followed by the second longest, and so on. In Scrabble a “bingo” is when you play all seven tiles in your rack, along with a letter on the board, to form an eight-letter word. What collection of 8 letters forms the most possible bingos? Solution: http://thinkpython2.com/code/anagram_sets.py.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

from time import time
def sorted_anagram(d):
    l = []
    for key in d:
        if len(d[key]) > 1:
            l.append((len(d[key]), d[key])) # 这是个由列表创建的元组？
    return sorted(l, reverse = True) # 倒序神马真折腾
def eight_letters(d, num):
    global length # 全局变量都用上了，就为了记录个最大值
    new_l = []
    for key in d:
        if len(key) == num and len(d[key]) > 1:
           new_l.append((len(d[key]), d[key]))
           if len(d[key]) >= length:
               length = len(d[key])
    return sorted(new_l)
def sorted_letters(word):
    list_word = sorted(list(word)) # 先把字符串打散为字符列表，然后排序
    reword =''.join(list_word) # 再把字符列表回粘成字符串
    return reword
def set_dict(fin):
    d = {}
    for line in fin:
        word = line.strip()
        reword = sorted_letters(word) # 打散重排相当关键，必须在建立字典时就做！！！
        if reword not in d:
            d[reword] = [word] # 字典的键已经不是单词，是纯粹的规律字符串
        else:
            d[reword].append(word) # 字典的键值才是词汇表里的单词
    return d
fin = open('words.txt')
length = 0
count = 0
start = time()
d = set_dict(fin)
for item in sorted_anagram(d):
    print(item)
    count += 1
print(count)
for item in eight_letters(d, 8):
    if item[0] == length:
        print(item)
end = time()
print(end - start)
# ......
# (2, ['abacas', 'casaba'])
# (2, ['aba', 'baa'])
# (2, ['aals', 'alas'])
# (2, ['aal', 'ala'])
# (2, ['aahed', 'ahead'])
# (2, ['aah', 'aha'])
# 10157 # 全体异构词
# (7, ['angriest', 'astringe', 'ganister', 'gantries', 'granites', 'ingrates', 'rangiest'])
# 异构词最多的8字母单词（共7个异构词）
# 0.6079998016357422
标签：, , , , , , , ,
还没有评论
2020-04
22

字典转元组

By xrspook @ 13:15:37 归类于: 扮IT

不搞复杂的，不用超纲的方法做感觉上很简单的事其实不简单。搞明白这个练习后，列表、字典、元组的相爱相杀我算是有点明白了。感谢那个我觉得过于复杂的参考答案，逼我折腾出了我自己的版本。

开心！居然习题1就用上了zip这个这章书最后才提到的大招。字典的键值对互换变得如此简单，我的脑洞又开大了。

Exercise 1: Write a function called most_frequent that takes a string and prints the letters in decreasing order of frequency. Find text samples from several different languages and see how letter frequency varies between languages. Compare your results with the tables at http://en.wikipedia.org/wiki/Letter_frequencies. Solution: http://thinkpython2.com/code/most_frequent.py. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

def most_frequent(sth):
    d = {}
    for letter in sth: # 字符串转为字符映射到频率的字典
        d[letter.lower()] = d.get(letter.lower(), 0) + 1 # 大写的你给我降为小写
    t = tuple(zip(d.values(), d.keys())) # 用zip互换字典的键和键值，生成元组（也可以生成列表，但生成新字典你会哭死）
    return sorted(t, reverse = True) # 以第一元素降序输出
sth = 'This chapter presents one more built-in type, the tuple, and then shows how lists, dictionaries, and tuples work together. I also present a useful feature for variable-length argument lists, the gather and scatter operators.'
t = most_frequent(sth)
for item in t:
    print(item) # 原汁原味输出元组好，因为空格不用''圈着都不知道那里有东西
# (33, ' ')
# (25, 'e')
# (23, 't')
# (16, 's')
# (15, 'r')
# (14, 'a')
# (11, 'o')
# (11, 'n')
# (10, 'i')
# (10, 'h')
# (9, 'l')
# (7, 'u')
# (7, 'p')
# (5, ',')
# (4, 'g')
# (4, 'd')
# (3, 'w')
# (3, 'f')
# (3, 'c')
# (2, 'm')
# (2, 'b')
# (2, '.')
# (2, '-')
# (1, 'y')
# (1, 'v')
# (1, 'k')
标签：, , , , , , ,
还没有评论
2020-04
22

写得过于简单了吧

By xrspook @ 9:29:12 归类于: 烂日记

我看的那个中文版Think Python 2的第11章，习题几乎可以这么说，没一道题的相关是做对的。有些东西，习题跟现在的Think Python 2英文版不一样，我觉得可能是英文版修正了某些东西。修正完以后，英文版并没有写什么更新说明之类的东西，但即便有，中文版翻译的人翻译完以后大概也不会时刻关注英文版有没有变动。同一道题的关联引用，这种东西怎么可以凭着感觉来呢？第一次遇到的时候，我云里雾里，因为按照中文版的那个说法，在某一章书的习题里根本找不到那个东西，但在我印象中，我的确是做过那些习题，到底是在哪一章书里的呢？另外一个版本的中文版那里的索引做对了。当我回看英文原版的时候，发现他们引用某个习题真的写得超简单，但是那些习题全部都是有超连接的，而且锚点到了某一章书的某个确切位置。所以我看的那个中文版之所以是现在这个样子，是不是因为他只是进行了文字上的翻译，还没有仔细研究那些超链接之类的东西呢？又或者在他翻译的时候，他并不是看着英文电子版，我是看英文纸质或者PDF等没有超链接之类的东西。电子版这种东西最让人觉得舒服的应该是哪怕你不标明是那是哪一章书的习题，你只是说那是习题几，只要你把超链接做上去，一切好办。

我看的那个中文版让我无语，但是英文原版也好不到哪里去。因为英文原版的作者脑洞实在太大，让人不知道如何言表。我觉得变成编程这东西，你得给出一个范围，然后再给出一个参考结果，才能让人有奔头。但是他只叫你这般测试，然后再给出一个参考的脚本。在不看脚本之前，如果我只是运行的脚本，而一开始的时候，我们设定的自定义参数不同，该如何判断读者自己的脚本写得对不对呢？有些参数他们完全可以先给出来，或者你会说，那岂不是成了一个提示。这又不是测验考试，重要的不是结果，而是如何整出那个结果，所以即便给你看到最终结果，你不懂过程，还是折腾不出来。英文原版的作者把那些条件参数写在他的脚本里面，而当你写自己的脚本的时候，你又清楚明白到你必须得设定那些参数，但是应该确定为什么呢？拿不准。这就会进入一个怪圈。首先是读者看着题目，自己解答，解答完以后测试，觉得没问题，打开参考答案运行，因为参数答案的结果跟读者得到的不一样，接着我们还得去仔细研究参考答案到底用了什么参数。然后把那个参数放在我们自己的脚本里再运行。对比两个运行的效果。这不是非常折腾吗？如果没有那个隐藏的参数，读者完全可以自己运行一遍，拿参考答案再运行一遍，对比结果就行了。我们最终要做到的不就是那个结果吗？当然，如果得出结果完全一致以后，还是建议大家再研究一下参考答案跟自己的脚本有什么区别，想一想这些区别会不会导致什么问题。

Think Python 2这本书，不只是习题参数设置很让人迷茫。在某些章节里，写得也不细致，比如说元组那一章。元组这个东西据说在其他编程语言里面是没有的。在介绍元组之前，已经说过字符串、列表以及字典。前面三个东西一个套一个，挺容易理解，但是加了元祖进去以后，世界就变得混沌了。因为列表字典元组这三个东西可以互相组合互相转化。有些可变，有些不可变。有些人可排序，有些无效。要真的说清楚元组，第12章书里面已经说到的东西以及已经举过的例子还不足以解决问题。如果他们不把所有东西都说一遍，起码他们得给我们一个列表。告诉我们这些东西可以控制元组，至于具体的使用方法，大家可以查手册。

或许当我把一整本书都看完以后，我的想法又不是我现在这个了。

标签：, , , , , , , , ,
还没有评论
1 2 3 4 5 6 7 … 9
© 2004 - 2024 我的天 | Theme by xrspook | Power by WordPress
Go to top