Mercurial > vec

annotate include/vec/impl/x86/sse2.h @ 15:e05c257c6a23

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
*: huge refactor, add many new x86 intrinsics and the like ALSO!! now intrinsics are enabled at runtime, depending on what is detected. altivec *should* still work but I only tested compiling it. the major version has been updated to 2.0 for this...
author Paper <paper@tflc.us>
date Wed, 20 Nov 2024 04:10:37 -0500
parents
children 41dd962abdd1
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
15
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
1 /**
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
2 * vec - a tiny SIMD vector library in C99
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
3 *
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
4 * Copyright (c) 2024 Paper
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
5 *
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
6 * Permission is hereby granted, free of charge, to any person obtaining a copy
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
7 * of this software and associated documentation files (the "Software"), to deal
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
8 * in the Software without restriction, including without limitation the rights
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
9 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
10 * copies of the Software, and to permit persons to whom the Software is
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
11 * furnished to do so, subject to the following conditions:
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
12 *
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
13 * The above copyright notice and this permission notice shall be included in all
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
14 * copies or substantial portions of the Software.
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
15 *
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
16 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
17 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
18 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
19 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
20 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
21 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
22 * SOFTWARE.
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
23 **/
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
24
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
25 #ifndef VEC_IMPL_X86_SSE2_H_
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
26 #define VEC_IMPL_X86_SSE2_H_
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
27
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
28 #define VEC_SSE2_OPERATION_8x16(op, sign) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
29 do { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
30 /* unpack and multiply */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
31 __m128i dst_even = _mm_##op##_epi16(vec1.sse, vec2.sse); \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
32 __m128i dst_odd = _mm_##op##_epi16(_mm_srli_epi16(vec1.sse, 8), _mm_srli_epi16(vec2.sse, 8)); \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
33 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
34 /* repack */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
35 return (v##sign##int8x16){ .sse = _mm_or_si128( \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
36 _mm_slli_epi16(dst_odd, 8), \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
37 _mm_srli_epi16(_mm_slli_epi16(dst_even, 8), 8) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
38 )}; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
39 } while (0)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
40
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
41 // shifting
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
42 #define VEC_SSE2_LSHIFT_8x16(sign) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
43 VEC_SSE2_OPERATION_8x16(sll, sign)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
44
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
45 #define VEC_SSE2_LSHIFT_16x8(sign) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
46 do { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
47 return (v##sign##int16x8){ .sse = _mm_sll_epi16(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
48 } while (0)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
49
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
50 #define VEC_SSE2_LSHIFT_32x4(sign) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
51 do { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
52 return (v##sign##int32x4){ .sse = _mm_sll_epi32(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
53 } while (0)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
54
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
55 #define VEC_SSE2_LSHIFT_64x2(sign) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
56 do { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
57 return (v##sign##int64x2){ .sse = _mm_sll_epi64(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
58 } while (0)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
59
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
60 #define VEC_SSE2_RSHIFT_8x16(sign, aORl) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
61 VEC_SSE2_OPERATION_8x16(sr##aORl, sign)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
62
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
63 #define VEC_SSE2_RSHIFT_16x8(sign, aORl) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
64 do { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
65 return (v##sign##int16x8){ .sse = _mm_sr##aORl##_epi16(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
66 } while (0)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
67
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
68 #define VEC_SSE2_RSHIFT_32x4(sign, aORl) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
69 do { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
70 return (v##sign##int32x4){ .sse = _mm_sr##aORl##_epi32(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
71 } while (0)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
72
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
73 #define VEC_SSE2_aRSHIFT_64x2(sign) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
74 do { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
75 return v##sign##int64x2_fallback_rshift(vec1, vec2); \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
76 } while (0)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
77
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
78 #define VEC_SSE2_lRSHIFT_64x2(sign) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
79 do { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
80 return (v##sign##int64x2){ .sse = _mm_srl_epi64(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
81 } while (0)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
82
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
83 #define VEC_SSE2_RSHIFT_64x2(sign, aORl) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
84 VEC_SSE2_##aORl##RSHIFT_64x2(sign)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
85
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
86 // shared between SSE2 variations
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
87 #define VEC_SSE2_MUL_8x16(sign) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
88 VEC_SSE2_OPERATION_8x16(mullo, sign)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
89
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
90 #define VEC_SSE2_MUL_16x8(sign) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
91 do { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
92 /* we have a real instruction for this */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
93 return (v##sign##int16x8){ .sse = _mm_mullo_epi16(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
94 } while (0)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
95
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
96 #define VEC_SSE2_MUL_32x4(sign) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
97 do { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
98 /* this was stolen from... somewhere :) */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
99 __m128i a13 = _mm_shuffle_epi32(vec1.sse, 0xF5); /* (-,a3,-,a1) */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
100 __m128i b13 = _mm_shuffle_epi32(vec2.sse, 0xF5); /* (-,b3,-,b1) */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
101 __m128i prod02 = _mm_mul_epu32(vec1.sse, vec2.sse); /* (-,a2*b2,-,a0*b0) */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
102 __m128i prod13 = _mm_mul_epu32(a13, b13); /* (-,a3*b3,-,a1*b1) */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
103 __m128i prod01 = _mm_unpacklo_epi32(prod02,prod13); /* (-,-,a1*b1,a0*b0) */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
104 __m128i prod23 = _mm_unpackhi_epi32(prod02,prod13); /* (-,-,a3*b3,a2*b2) */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
105 return (v##sign##int32x4) { .sse = _mm_unpacklo_epi64(prod01, prod23) }; /* (ab3,ab2,ab1,ab0) */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
106 } while (0)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
107
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
108 #define VEC_SSE2_MUL_64x2(sign) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
109 do { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
110 __m128i ac = _mm_mul_epu32(vec1.sse, vec2.sse); /* ac = (vec1 & UINT32_MAX) * (vec2 & UINT32_MAX); */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
111 __m128i b = _mm_srli_epi64(vec1.sse, 32); /* b = vec1 >> 32; */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
112 __m128i bc = _mm_mul_epu32(b, vec2.sse); /* bc = b * (vec2 & UINT32_MAX); */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
113 __m128i d = _mm_srli_epi64(vec2.sse, 32); /* d = vec2 >> 32; */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
114 __m128i ad = _mm_mul_epu32(vec1.sse, d); /* ad = (vec1 & UINT32_MAX) * d; */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
115 __m128i hi = _mm_add_epi64(bc, ad); /* hi = bc + ad; */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
116 hi = _mm_slli_epi64(hi, 32); /* hi <<= 32; */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
117 return (v##sign##int64x2) { .sse = _mm_add_epi64(hi, ac) }; /* return ac + hi; */ \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
118 } while (0)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
119
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
120 #define VEC_SSE2_DEFINE_OPERATIONS_SIGN(sign, bits, size) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
121 static v##sign##int##bits##x##size v##sign##int##bits##x##size##_sse2_load_aligned(const sign##int##bits##_t in[size]) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
122 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
123 return (v##sign##int##bits##x##size) { .sse = _mm_load_si128((const __m128i *)in) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
124 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
125 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
126 static v##sign##int##bits##x##size v##sign##int##bits##x##size##_sse2_load(const sign##int##bits##_t in[size]) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
127 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
128 return (v##sign##int##bits##x##size) { .sse = _mm_loadu_si128((const __m128i *)in) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
129 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
130 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
131 static void v##sign##int##bits##x##size##_sse2_store_aligned(v##sign##int##bits##x##size vec, sign##int##bits##_t out[size]) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
132 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
133 _mm_store_si128((__m128i *)out, vec.sse); \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
134 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
135 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
136 static void v##sign##int##bits##x##size##_sse2_store(v##sign##int##bits##x##size vec, sign##int##bits##_t out[size]) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
137 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
138 _mm_storeu_si128((__m128i *)out, vec.sse); \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
139 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
140 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
141 static v##sign##int##bits##x##size v##sign##int##bits##x##size##_sse2_add(v##sign##int##bits##x##size vec1, v##sign##int##bits##x##size vec2) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
142 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
143 return (v##sign##int##bits##x##size) { .sse = _mm_add_epi##bits(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
144 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
145 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
146 static v##sign##int##bits##x##size v##sign##int##bits##x##size##_sse2_sub(v##sign##int##bits##x##size vec1, v##sign##int##bits##x##size vec2) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
147 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
148 return (v##sign##int##bits##x##size) { .sse = _mm_sub_epi##bits(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
149 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
150 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
151 static v##sign##int##bits##x##size v##sign##int##bits##x##size##_sse2_mul(v##sign##int##bits##x##size vec1, v##sign##int##bits##x##size vec2) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
152 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
153 VEC_SSE2_MUL_##bits##x##size(sign); \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
154 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
155 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
156 static v##sign##int##bits##x##size v##sign##int##bits##x##size##_sse2_and(v##sign##int##bits##x##size vec1, v##sign##int##bits##x##size vec2) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
157 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
158 return (v##sign##int##bits##x##size) { .sse = _mm_and_si128(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
159 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
160 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
161 static v##sign##int##bits##x##size v##sign##int##bits##x##size##_sse2_or(v##sign##int##bits##x##size vec1, v##sign##int##bits##x##size vec2) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
162 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
163 return (v##sign##int##bits##x##size) { .sse = _mm_or_si128(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
164 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
165 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
166 static v##sign##int##bits##x##size v##sign##int##bits##x##size##_sse2_xor(v##sign##int##bits##x##size vec1, v##sign##int##bits##x##size vec2) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
167 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
168 return (v##sign##int##bits##x##size) { .sse = _mm_xor_si128(vec1.sse, vec2.sse) }; \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
169 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
170 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
171 static v##sign##int##bits##x##size v##sign##int##bits##x##size##_sse2_lshift(v##sign##int##bits##x##size vec1, vuint##bits##x##size vec2) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
172 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
173 VEC_SSE2_LSHIFT_##bits##x##size(sign); \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
174 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
175 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
176 static v##sign##int##bits##x##size v##sign##int##bits##x##size##_sse2_rshift(v##sign##int##bits##x##size vec1, vuint##bits##x##size vec2) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
177 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
178 VEC_SSE2_RSHIFT_##bits##x##size(sign, a); \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
179 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
180 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
181 static v##sign##int##bits##x##size v##sign##int##bits##x##size##_sse2_lrshift(v##sign##int##bits##x##size vec1, vuint##bits##x##size vec2) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
182 { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
183 VEC_SSE2_RSHIFT_##bits##x##size(sign, l); \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
184 } \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
185 \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
186 static v##sign##int##bits##x##size##_impl v##sign##int##bits##x##size##_impl_sse2 = { \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
187 .load_aligned = v##sign##int##bits##x##size##_sse2_load_aligned, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
188 .load = v##sign##int##bits##x##size##_sse2_load, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
189 .store_aligned = v##sign##int##bits##x##size##_sse2_store_aligned, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
190 .store = v##sign##int##bits##x##size##_sse2_store, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
191 .add = v##sign##int##bits##x##size##_sse2_add, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
192 .sub = v##sign##int##bits##x##size##_sse2_sub, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
193 .mul = v##sign##int##bits##x##size##_sse2_mul, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
194 .and = v##sign##int##bits##x##size##_sse2_and, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
195 .or = v##sign##int##bits##x##size##_sse2_or, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
196 .xor = v##sign##int##bits##x##size##_sse2_xor, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
197 .lshift = v##sign##int##bits##x##size##_sse2_lshift, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
198 .rshift = v##sign##int##bits##x##size##_sse2_rshift, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
199 .lrshift = v##sign##int##bits##x##size##_sse2_lrshift, \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
200 };
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
201
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
202 #define VEC_SSE2_DEFINE_OPERATIONS(bits, size) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
203 VEC_SSE2_DEFINE_OPERATIONS_SIGN( , bits, size) \
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
204 VEC_SSE2_DEFINE_OPERATIONS_SIGN(u, bits, size)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
205
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
206 // SSE is *only* 128-bit
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
207 VEC_SSE2_DEFINE_OPERATIONS(8, 16)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
208 VEC_SSE2_DEFINE_OPERATIONS(16, 8)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
209 VEC_SSE2_DEFINE_OPERATIONS(32, 4)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
210 VEC_SSE2_DEFINE_OPERATIONS(64, 2)
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
211
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
212 #undef VEC_SSE2_DEFINE_OPERATIONS
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
213 #undef VEC_SSE2_DEFINE_OPERATIONS_SIGN
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
214 #undef VEC_SSE2_MUL_8x16
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
215 #undef VEC_SSE2_MUL_16x8
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
216 #undef VEC_SSE2_MUL_32x4
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
217 #undef VEC_SSE2_MUL_64x2
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
218 #undef VEC_SSE2_OPERATION_8x16
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
219 #undef VEC_SSE2_LSHIFT_8x16
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
220 #undef VEC_SSE2_LSHIFT_16x8
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
221 #undef VEC_SSE2_LSHIFT_32x4
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
222 #undef VEC_SSE2_LSHIFT_64x2
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
223 #undef VEC_SSE2_RSHIFT_8x16
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
224 #undef VEC_SSE2_RSHIFT_16x8
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
225 #undef VEC_SSE2_RSHIFT_32x4
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
226 #undef VEC_SSE2_aRSHIFT_64x2
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
227 #undef VEC_SSE2_lRSHIFT_64x2
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
228 #undef VEC_SSE2_RSHIFT_64x2
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
229
e05c257c6a23 *: huge refactor, add many new x86 intrinsics and the like
Paper <paper@tflc.us>
parents:
diff changeset
230 #endif /* VEC_IMPL_X86_SSE2_H_ */