Posterior a observar en diferentes repositorios y sitios webs al concluir hemos hallado la respuesta que te compartimos pronto.
Solución:
Aquí hay una solución vectrorizada rápida y que ahorra memoria, que usa el método sklearn.feature_extraction.text.CountVectorizer:
from sklearn.feature_extraction.text import CountVectorizer
vect = CountVectorizer(min_df=1, lowercase=True)
X = vect.fit_transform(df['Employer'])
cols_emp = vect.get_feature_names()
X = vect.fit_transform(df['Description'])
cols_desc = vect.get_feature_names()
common_cols_idx = [i for i,col in enumerate(cols_desc) if col in cols_emp]
df['Match'] = (X.toarray()[:, common_cols_idx] == 1).any(1)
Fuente DF:
In [259]: df
Out[259]:
Date Description Amount AutoNumber Employer
0 3/17/2015 WW120 TFR?FR xxx8690 140.00 49246 Cansel Survey Equipment
2 3/13/2015 JX154 TFR?FR xxx8690 150.00 49246 Cansel Survey Equipment
5 3/6/2015 CANSEL SURVEY E PAY 1182.08 49246 Cansel Survey Equipment
9 3/2/2015 UE200 TFR?FR xxx8690 180.00 49246 Cansel Survey Equipment
10 2/27/2015 JH401 TFR?FR xxx8690 400.00 49246 Cansel Survey Equipment
11 2/27/2015 CANSEL SURVEY E PAY 555.62 49246 Cansel Survey Equipment
12 2/25/2015 HU204 TFR?FR xxx8690 200.00 49246 Cansel Survey Equipment
13 2/23/2015 UQ263 TFR?FR xxx8690 102.00 49246 Cansel Survey Equipment
14 2/23/2015 UT460 TFR?FR xxx8690 200.00 49246 Cansel Survey Equipment
15 2/20/2015 CANSEL SURVEY E PAY 1222.05 49246 Cansel Survey Equipment
17 2/17/2015 UO414 TFR?FR xxx8690 250.00 49246 Cansel Survey Equipment
19 2/11/2015 HI540 TFR?FR xxx8690 130.00 49246 Cansel Survey Equipment
20 2/11/2015 HQ010 TFR?FR xxx8690 177.00 49246 Cansel Survey Equipment
21 2/10/2015 WU455 TFR?FR xxx8690 200.00 49246 Cansel Survey Equipment
22 2/6/2015 JJ500 TFR?FR xxx8690 301.00 49246 Cansel Survey Equipment
23 2/6/2015 CANSEL SURVEY E PAY 1182.08 49246 Cansel Survey Equipment
24 2/5/2015 IR453 TFR?FR xxx8690 168.56 49246 Cansel IR453
26 2/2/2015 RQ574 TFR?FR xxx8690 500.00 49246 Cansel Survey Equipment
27 2/2/2015 UT022 TFR?FR xxx8690 850.00 49246 Cansel Survey Equipment
28 12/31/2014 HU521 TFR?FR xxx8690 950.17 49246 Cansel Survey HU521
Resultado:
In [261]: df
Out[261]:
Date Description Amount AutoNumber Employer Match
0 3/17/2015 WW120 TFR?FR xxx8690 140.00 49246 Cansel Survey Equipment False
2 3/13/2015 JX154 TFR?FR xxx8690 150.00 49246 Cansel Survey Equipment False
5 3/6/2015 CANSEL SURVEY E PAY 1182.08 49246 Cansel Survey Equipment True
9 3/2/2015 UE200 TFR?FR xxx8690 180.00 49246 Cansel Survey Equipment False
10 2/27/2015 JH401 TFR?FR xxx8690 400.00 49246 Cansel Survey Equipment False
11 2/27/2015 CANSEL SURVEY E PAY 555.62 49246 Cansel Survey Equipment True
12 2/25/2015 HU204 TFR?FR xxx8690 200.00 49246 Cansel Survey Equipment False
13 2/23/2015 UQ263 TFR?FR xxx8690 102.00 49246 Cansel Survey Equipment False
14 2/23/2015 UT460 TFR?FR xxx8690 200.00 49246 Cansel Survey Equipment False
15 2/20/2015 CANSEL SURVEY E PAY 1222.05 49246 Cansel Survey Equipment True
17 2/17/2015 UO414 TFR?FR xxx8690 250.00 49246 Cansel Survey Equipment False
19 2/11/2015 HI540 TFR?FR xxx8690 130.00 49246 Cansel Survey Equipment False
20 2/11/2015 HQ010 TFR?FR xxx8690 177.00 49246 Cansel Survey Equipment False
21 2/10/2015 WU455 TFR?FR xxx8690 200.00 49246 Cansel Survey Equipment False
22 2/6/2015 JJ500 TFR?FR xxx8690 301.00 49246 Cansel Survey Equipment False
23 2/6/2015 CANSEL SURVEY E PAY 1182.08 49246 Cansel Survey Equipment True
24 2/5/2015 IR453 TFR?FR xxx8690 168.56 49246 Cansel IR453 True
26 2/2/2015 RQ574 TFR?FR xxx8690 500.00 49246 Cansel Survey Equipment False
27 2/2/2015 UT022 TFR?FR xxx8690 850.00 49246 Cansel Survey Equipment False
28 12/31/2014 HU521 TFR?FR xxx8690 950.17 49246 Cansel Survey HU521 True
Algunas explicaciones:
In [266]: cols_desc
Out[266]:
['cansel',
'fr',
'hi540',
'hq010',
'hu204',
'hu521',
'ir453',
'jh401',
'jj500',
'jx154',
'pay',
'rq574',
'survey',
'tfr',
'ue200',
'uo414',
'uq263',
'ut022',
'ut460',
'wu455',
'ww120',
'xxx8690']
In [267]: cols_emp
Out[267]: ['cansel', 'equipment', 'hu521', 'ir453', 'survey']
In [268]: common_cols_idx = [i for i,col in enumerate(cols_desc) if col in cols_emp]
In [269]: common_cols_idx
Out[269]: [0, 5, 6, 12]
In [270]: X.toarray()
Out[270]:
array([[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1],
[0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1],
[0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
[0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1],
[0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1],
[0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1]], dtype=int64)
In [271]: X.toarray()[:, common_cols_idx]
Out[271]:
array([[0, 0, 0, 0],
[0, 0, 0, 0],
[1, 0, 0, 1],
[0, 0, 0, 0],
[0, 0, 0, 0],
[1, 0, 0, 1],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[1, 0, 0, 1],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[1, 0, 0, 1],
[0, 0, 1, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 1, 0, 0]], dtype=int64)
In [272]: X.toarray()[:, common_cols_idx] == 1
Out[272]:
array([[False, False, False, False],
[False, False, False, False],
[ True, False, False, True],
[False, False, False, False],
[False, False, False, False],
[ True, False, False, True],
[False, False, False, False],
[False, False, False, False],
[False, False, False, False],
[ True, False, False, True],
[False, False, False, False],
[False, False, False, False],
[False, False, False, False],
[False, False, False, False],
[False, False, False, False],
[ True, False, False, True],
[False, False, True, False],
[False, False, False, False],
[False, False, False, False],
[False, True, False, False]], dtype=bool)
In [273]: (X.toarray()[:, common_cols_idx] == 1).any(1)
Out[273]: array([False, False, True, False, False, True, False, False, False, True, False, False, False, False, False, True, True, Fals
e, False, True], dtype=bool)
Aquí hay una solución, que divide el texto en conjuntos de minúsculas y usa la intersección de conjuntos para cada fila:
In [160]: x['Match'] = x.Description.str.lower().str.split().map(set).to_frame('desc')
...: .apply(lambda r: (x.Employer.str.lower().str.split().map(set) & r.desc).any(),
...: axis=1)
...:
In [161]: x
Out[161]:
Date Description Amount AutoNumber Employer Match
0 3/17/2015 WW120 TFR?FR xxx8690 140.00 49246 Cansel Survey Equipment False
2 3/13/2015 JX154 TFR?FR xxx8690 150.00 49246 Cansel Survey Equipment False
5 3/6/2015 CANSEL SURVEY E PAY 1182.08 49246 Cansel Survey Equipment True
9 3/2/2015 UE200 TFR?FR xxx8690 180.00 49246 Cansel Survey Equipment False
10 2/27/2015 JH401 TFR?FR xxx8690 400.00 49246 Cansel Survey Equipment False
11 2/27/2015 CANSEL SURVEY E PAY 555.62 49246 Cansel Survey Equipment True
12 2/25/2015 HU204 TFR?FR xxx8690 200.00 49246 Cansel Survey Equipment False
13 2/23/2015 UQ263 TFR?FR xxx8690 102.00 49246 Cansel Survey Equipment False
14 2/23/2015 UT460 TFR?FR xxx8690 200.00 49246 Cansel Survey Equipment False
15 2/20/2015 CANSEL SURVEY E PAY 1222.05 49246 Cansel Survey Equipment True
17 2/17/2015 UO414 TFR?FR xxx8690 250.00 49246 Cansel Survey Equipment False
19 2/11/2015 HI540 TFR?FR xxx8690 130.00 49246 Cansel Survey Equipment False
20 2/11/2015 HQ010 TFR?FR xxx8690 177.00 49246 Cansel Survey Equipment False
21 2/10/2015 WU455 TFR?FR xxx8690 200.00 49246 Cansel Survey Equipment False
22 2/6/2015 JJ500 TFR?FR xxx8690 301.00 49246 Cansel Survey Equipment False
23 2/6/2015 CANSEL SURVEY E PAY 1182.08 49246 Cansel Survey Equipment True
24 2/5/2015 IR453 TFR?FR xxx8690 168.56 49246 Cansel Survey Equipment False
26 2/2/2015 RQ574 TFR?FR xxx8690 500.00 49246 Cansel Survey Equipment False
27 2/2/2015 UT022 TFR?FR xxx8690 850.00 49246 Cansel Survey Equipment False
28 12/31/2014 HU521 TFR?FR xxx8690 950.17 49246 Cansel Survey Equipment False
PD: es bastante lento ya que está usando no vectorizado .apply(..., axis=1) método
Aquí hay una solución legible usando un individuo search_func:
def search_func(row):
matches = [test_value in row["Description"].lower()
for test_value in row["Text_Search"]]
if any(matches):
return "Yes"
else:
return "No"
Esta función luego se aplica por filas:
# create example data
df = pd.DataFrame("Description": ["CANSEL SURVEY E PAY", "JX154 TFR?FR xxx8690"],
"Employer": ["Cansel Survey Equipment", "Cansel Survey Equipment"])
print(df)
Description Employer
0 CANSEL SURVEY E PAY Cansel Survey Equipment
1 JX154 TFR?FR xxx8690 Cansel Survey Equipment
# create text searches and match column
df["Text_Search"] = df["Employer"].str.lower().str.split()
df["Match"] = df.apply(search_func, axis=1)
# show result
print(df)
Description Employer Text_Search Match
0 CANSEL SURVEY E PAY Cansel Survey Equipment [cansel, survey, equipment] Yes
1 JX154 TFR?FR xxx8690 Cansel Survey Equipment [cansel, survey, equipment] No
Comentarios y puntuaciones del post
Tienes la opción de añadir valor a nuestra información añadiendo tu experiencia en los comentarios.
¡Haz clic para puntuar esta entrada!
(Votos: 0 Promedio: 0)